PE-ARP
Port-Enhanced ARP (Address Resolution Protocol)
- End-Host Source Port Range Management
- Network Communication Protocol for IP Address and Port Information
- Mapping IPv4 Packets to Network End-Hosts
- Enhanced DNS Look-Up Service
PE-ARP is a new method for sharing IPv4 addresses, while maintaining the end-to-end principle of the Internet. Unlike NAT, each end-host shares a public IP address, and distinguishes itself from the other hosts by having a unique port range that it uses. For example, two hosts (A and B) can share the IP address 198.108.63.2 if host A was limited to ports 1000-1999, and host B was limited to ports 2000-2999.
PE-ARP enabled end-hosts are able to communicate to external hosts, provided a protocol with ports is used (i.e TCP or UDP).
For more information on PE-ARP, see our white paper
End-Host Source Port Range Management
The first component of the system is the end-host port range management module. Each application on a network end-host requests and obtains local port values from the operating system. These values can either be explicitly requested by the application or randomly assigned by the Operating System. To enable IPv4 address sharing, the range of local port values that is used by each end-host must be limited. In our implementation, this is done by setting the end-host's port range in the kernel via the sysctl command.
End-Host Configuration
Each host will have to run our modified kernel (patch). The hosts must have a unique port range that they will use. An example configuration of two hosts is shown below. Host A is using 198.108.63.2:1000-1999, and Host B is using 198.108.63.2:2000-2999 (ports are inclusive).
#host A /etc/network/interfaces file auto lo iface lo inet loopback # Loopback device auto eth0 # The primary network interface iface eth0 inet static address 198.108.63.2 network 198.108.63.0 netmask 255.255.255.0 gateway 198.108.63.1 pre-up sysctl -w net.ipv4.ip_local_port_range="1000 1999"
#host B /etc/network/interfaces file auto lo iface lo inet loopback # Loopback device auto eth0 # The primary network interface iface eth0 inet static address 198.108.63.2 network 198.108.63.0 netmask 255.255.255.0 gateway 198.108.63.1 pre-up sysctl -w net.ipv4.ip_local_port_range="2000 2999"
Notice that the only difference between these configuration files is the port range for the hosts.
It is also possible to use dhcp to assign port ranges; see PE-ARP Enabled DHCP below.
Network Communication Protocol for IP Address and Port Information
The second major component of the system is the network communication protocol to share local port range information with the local network. The ultimate goal of this protocol is to provide every host on the network with enough information to send IPv4 packets to the correct physical host. We have extended the basic ARP protocol to include both IP addresses and port ranges so it can be used to locate PE-ARP hosts. When the local router needs to know what host is using a particular port, the router sends out a broadcast request for the information (PE-ARP_REQUEST). The correct host responds (PE-ARP_REPLY), providing its hardware address and full local port range.
Mapping IPv4 Packets to Network End-Hosts
The third component of the overall PE-ARP architecture is the extension to the local ARP table itself. When a packet arrives from the Internet to the local router, the router has to determine the packet's end-host destination. In current networks, the local router stores a mapping between IP addresses and physical MAC addresses. However, we now allow multiple end-hosts to share the same IPv4 address, so this information is no longer sufficient to uniquely identify the destination end-host for a packet. Instead, a modified table is used that employs both IP and port information to determine the MAC address of the end-host for which the packets are intended. In addition to the IP address, this table includes two port values, which are used to specify the range of ports that are associated with a given host.
pe-arp-hostA:~$ arp -n Address HWtype HWaddress Flags Mask Iface 192.35.162.2:2000-2999 ether 00:0c:29:9e:8c:ee C eth1 192.35.162.2:3000-3999 ether 00:0c:29:39:44:ab C eth1 192.35.162.2:0-0 ether 00:12:7f:c4:38:d3 C eth0
Bridge Configuration
In a bridged configuration, a PE-ARP aware bridge acts as a translation point for the PE-ARP hosts. An example is shown in the figure below.
An example configuration to set up the bridge (or Ferry) is shown below. This script should be run at startup.
/sbin/ifconfig eth1 0.0.0.0 #bring up two physical interfaces /sbin/ifconfig eth0 0.0.0.0 #without ip addresses /usr/sbin/brctl addbr br0 #create bridge /usr/sbin/brctl addif br0 eth0 #add interfaces to bridge /usr/sbin/brctl addif br0 eth1 /sbin/ifconfig br0 inet 198.108.63.2 netmask 255.255.255.0 #assign shared ip to bridge /sbin/route add default gw 198.108.63.1 #add route to gateway sysctl -w net.ipv4.ip_local_port_range="1 99" #set port range for this host to 1-99
The bridge (ferry) will need to run our modified bridge kernel (patch).
Gateway Configuration
In a gatewayed configuration, a PE-ARP aware router is the translation point for the PE-ARP hosts. An example is shown in the figure below.
An example configuration to set up the gateway is shown below.
#Gateway /etc/network/interfaces file auto lo iface lo inet loopback #Loopback device auto eth0 iface eth0 inet static #outward facing interface address 198.108.63.2 #(non PE-ARP aware) network 198.108.63.0 netmask 255.255.255.0 gateway 198.108.63.1 auto eth1 iface eth1 inet static #inward facing interface address 192.35.162.1 #(PE-ARP aware end-hosts) network 192.35.162.0 netmask 255.255.255.248 pre-up sysctl -w net.ipv4.ip_local_port_range="1 65536" pre-up echo 1 > /proc/sys/net/ipv4/ip_forward #become a router
PE-ARP Enabled DHCP
We have modified ISC dhcp server and client to support a port range option. Port ranges, like IP addresses, can either be allocated out of a pool or assigned statically.
dhcp-4.1.1.diff
Server
Static port range assignments are given in the corresponding "host" declaration of the dhcpd.conf file.
host anzi {
hardware ethernet 00:0a:e4:3c:d7:c3;
fixed-address 192.168.3.57;
option port-range "10000-11000";
}
Dynamic port ranges are allocated out of a pool.
Since all clients using a port range from a given pool
share the same IP address, the "range" declaration only includes one IP address.
The number of ports to hand out to each client is given in a "port-chunksize"
declaration.
# This is a set of PE-ARP hosts with a single IP address and multiple port
# ranges. The ranges start at 12000 and have 1000 ports each.
subnet 192.168.0.0 netmask 255.255.252.0 {
range 192.168.3.57:12000~15000;
port-chunksize 1000;
}
Client
The dhclient.conf file must specify a request for the "port-range" option, either in the "request" declaration or as an "also request":
request subnet-mask, broadcast-address, time-offset, routers,
domain-name, domain-name-servers, domain-search, host-name,
netbios-name-servers, netbios-scope, interface-mtu,
rfc3442-classless-static-routes, ntp-servers, port-range;
or
also request port-range;
There is also an exit hook script that must be installed to handle the port-range option. This will normally be installed in /etc/dhcp3/dhclient-exit-hooks.d. If you are using Network Manager, it will have to be modified to run exit hooks.
Enhanced DNS Look-Up Service
In implementing PE-ARP we have leveraged DNS SRV records for locating services on PE-ARP hosts. An SRV record maps a domain name and a service name to a canonical domain name and a port number. A PE-ARP service host can publish its service ports in this way. One problem with this approach is that not all client applications are capable of using SRV records in place of well-known ports.
With PE-ARP, all services are required to be published through the DNS system. No changes need to be made to either the DNS protocol or a DNS implementation as SRV records are well understood and implemented.
SRV queries by default using libpearp
libpearp replaces the system's default getaddrinfo function with one that makes DNS SRV queries in order to find the appropriate port numbers for services on different hosts. Userspace applications that use getaddrinfo will automatically benefit if they pass a service name to getaddrinfo.
By default, getaddrinfo looks in your system's /etc/services file to find out which port to connect on for any given service. libpearp's getaddrinfo function makes an SRV query, using DNS to find the correct port (and in turn the correct PE-ARP enabled host). libpearp will fall back to the system's getaddrinfo function if it does not find a suitable SRV entry or is not provided with a valid service name.
libpearp relies only on libraries internal to glibc, namely libresolv, libanl, and libdl. Any system with glibc should have no quarrel with libpearp.
Installation
To install libpearp, cd into the libpearp directory and do:$ make $ su $ make installThen, you must add the path to libpearp to the LD_PRELOAD environment variable. For example, to set LD_PRELOAD for the current session, you might do:
$ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/libpearp.soOr you can add this to your /etc/profile or ~/.bashrc:
export LD_PRELOAD=$LD_PRELOAD:/usr/lib/libpearp.so
Modifying Applications to Use SRV Queries
The template for getaddrinfo is the following:
int
getaddrinfo(const char *node, const char *service,
const struct addrinfo *hints, struct addrinfo **res);
Many applications that use getaddrinfo treat the service parameter as an ascii string representation of a desired port number because that's a valid thing to do according to the getaddrinfo specification. That means that using libpearp, many applications will not make SRV queries out of the box. To get these applications to make SRV queries for ports, it is necessary to modify them to pass a service name as the service parameter (not strPort or NULL). Then, if an SRV record is available, it will be used to fill in the sockaddr structure.
Privileged Applications
Because it relies on LD_PRELOAD, libpearp only replaces getaddrinfo for userspace applications. If you wish to use libpearp's getaddrinfo function in privileged applications or applications that are running setuid, you must make two changes to the application:
First, include pearp/pearp.h in files where you use getaddrinfo, like so:#include <pearp/pearp.h>
Then, replace all occurrences of getaddrinfo with calls to pearp_getaddrinfo.
Finally, rebuild the application, linking against libpearp:$ gcc ... -lpearp