Recently, I have found myself with two independent internet feeds. The nice side effect of having two feeds is you can configure multipath routing, load balancing, failover, and dead gateway detection. However, for a individual setup there isn’t much value in configuring all that is necessary for the latter items. Nevertheless, configuring multipath routing can be quite valuable, allowing select traffic to travel over an intentionally underutilized link for reduced latency, increased bandwidth, or what have you.

Assuming you already have a Linux router configured for one interface and said router is now plugged into a second Internet feed, configuration is surprisingly easy.

First, disable /proc/sys/net/ipv4/conf/$IFACE/rp_filter as it will give you grief later. $IFACE represents the feed you want to selectively balance or route to. In other words, the feed that is not currently your default route.

# echo 0 >/proc/sys/net/ipv4/conf/eth3/rp_filter

Next, use ip from the iproute2 package to create a new routing table. I like to name the table based on the last octet of the IP address of its specified default route. The fwmark is a marker managed inside the kernel itself that can be applied to a packet by iproute2 tools or using iptables. The packet itself is not modified.

# ip route add table 4 default via 1.2.3.4 dev eth3
# ip rule add fwmark 0×4 table 4
# ip route flush cache

Above, we created our new routing table, with a default route of 1.2.3.4 and explicit interface dev eth3. Next, we created a rule that will assign any traffic with our marker, 0×4 in hex in the example above, to routing table identified as 4. It’s important to note that there is a difference between specifying, say, fwmark 100 and fwmark 0x100. The former is converted to hex internally whereas the latter remains 0×100 as you’d expect. Simply specifying your fwmark as hex by including the leading 0x is the easiest way of keeping things straight.

Next, it’s time to add our iptables magic to mark the flows we want to be directed out our alternate route. (You could also accomplish this with a static route, if some specific internal host should always use the alternate route, but that’s highly inflexible.) The usual iptables syntax applies.

# iptables -t mangle -A PREROUTING -p tcp -s 192.168.0.228 \
  --dport 80 -j MARK --set-mark 0×4

Above, all traffic from host 192.168.0.228 to any host on port 80 passing through the Linux router will be marked accordingly. Once marked, it will be shuffled into the appropriate routing table, 4, defined earlier.

# iptables -t mangle -vnL | grep 192.168.0.228

If you generate some traffic on the host in question, you should see hits on the new mark rule.

Finally, let’s add an SNAT rule for iptables. If the IPs you’ll be operating over are all routable publicly, this isn’t necessary. For unroutable addresses, you’ll probably want this so outbound packets are rewritten to have the correct source address of your external address. Substitute MASQUERADE as necessary if your public IP is dynamic.

# iptables -t nat -A POSTROUTING -o eth3 -j SNAT --to-source 1.2.3.3

Of course, it’s assumed you already have a working iptables configuration and that everything already works correctly with a single Internet link. Configuring all that is left as an exercise for the reader.

If anyone’s found a way to do this using only the ROUTE target available for Netfilter instead of creating a dedicated routing table, I’d love to hear about it. I tried for some time to no avail, as it wasn’t possible under 2.6.15 to use SNAT and ROUTE at the same time — ROUTE always won, giving me packets with nonroutable source addresses.

Martin Brown’s Guide to IP Layer Network Administration with Linux has a section on multipath routing that might be useful if the above doesn’t do what you need. Or you may want to do full load balancing and might find Hisham Mardam Bey’s article on Load Balancing Across Multiple Links helpful. Additionally, if you want to load balance across multiple gateways and can’t assign multiple IP addresses to a server, you might try this novel approach suggested by Philip Craig instead.