While working on multi-homing my ASN network earlier to add Vultr as my secondary upstream, I ran into an issue where my traffic coming out of my Vultr BGP session was seen as coming from Vultr’s IP instead of my own static IP prefix.
This caused a bunch of problems with my secondary DNS servers for TheLittleHost, as they suddenly did not recognise my hidden master server because its IP (now the Vultr IP) was not the IP they were expecting.
A curl from any of the servers behind the router would return:
server $ curl https://ifconfig.co
<vultr ip address>
The Problem
In standard VPS setups where we’re using the VPS as a router into a private network (like my home), we often want to let the private network access the internet, and to do so it is usually common practice to add a blanket MASQUERADE (NAT) rule
iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
This rule tells the interface enp1s0 that anything that is going out of it will have its source address rewritten to the interface’s public IP.
This is so that when traffic goes out, any responses will know how to reach back into your private network by first stopping by the router where it left (using its IP address). Once it reaches back to the router, it is then able to rewrite it and forward the packet into the internal network.
However, in our BGP multi-homed network, we already own our own public IP addresses and our servers behind the routers are assigned a public address that should be publicly accessible.
If we allow the packet source addresses to be masqueraded, we would lose the whole point of BGP and having our own addresses.
Visualising the Problem
So what is actually happening?
sequenceDiagram
participant Host as Server (x.x.x.215)
participant VMRouter as VM Router
participant Vultr as Vultr VPS (Blanket MASQUERADE)
participant Internet as The Internet (GitHub)
Host->>VMRouter: TCP SYN (Src: x.x.x.215, Dst: GitHub)
VMRouter->>Vultr: Forward over WG (Src: x.x.x.215, Dst: GitHub)
Note over Vultr: BLANKET MASQUERADE HITS!<br/>Source IP is rewritten
Vultr->>Internet: Forward to GitHub (Src: Vultr Public IP, Dst: GitHub)
Internet->>Vultr: Reply (Src: GitHub, Dst: Vultr Public IP)
Note over Vultr: Vultr un-NATs the packet<br/>and sends back over WG
Vultr->>VMRouter: Forward over WG (Src: GitHub, Dst: x.x.x.215)
Note over VMRouter,Host: PROBLEM: If BGP path changes, or asymmetry occurs,<br/>connmark and state tables get confused.<br/>Packets can be dropped.
So to fix this, we need to our Vultr VPS to exclude our IP prefix from any POSTROUTING masquerade activity.
Instead of changing the existing MASQUERADE rule, we can just add an ACCEPT rule at the top of the POSTROUTING chain so that it would skip the MASQUERADE if it matches our BGP subnet.
iptables -t nat -I POSTROUTING 1 -o enp1s0 -s x.x.x.0/24 -j ACCEPT
# existing rule
# iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
Now with this change, end-to-end public routing is preserved.
And this is how it looks like after we skip masquerade.
sequenceDiagram
participant Host as Server (x.x.x.215)
participant VMRouter as VM Router
participant Vultr as Vultr VPS (Fixed NAT)
participant Internet as The Internet (GitHub)
Host->>VMRouter: TCP SYN (Src: x.x.x.215, Dst: GitHub)
VMRouter->>Vultr: Forward over WG (Src: x.x.x.215, Dst: GitHub)
Note over Vultr: NEW ACCEPT RULE HITS!<br/>"Source is x.x.x.0/24?"<br/>YES -> Skip MASQUERADE
Vultr->>Internet: Forward to GitHub (Src: x.x.x.215, Dst: GitHub)
Internet->>Vultr: Reply (Src: GitHub, Dst: x.x.x.215)
Note over Vultr: BGP routing brings the reply<br/>directly to the right BGP IP
Vultr->>VMRouter: Forward over WG (Src: GitHub, Dst: x.x.x.215)
VMRouter->>Host: Packet Delivered
Note over Host,Internet: SUCCESS: End-to-end public routing is preserved.<br/>connmark and asymmetric paths work perfectly!
So something to take note when working with routing between a private network and a public one. Especially if you’re routing with your own IP prefix.