After planning for this for a while, I finally found some time yesterday to finish hooking up my network to support multi-home.
One of the biggest obstacle that got me procrastinating was configuring the wireguard between my Home and my Datacenter. I needed to make sure that regardless of which direction traffic comes from, either one of the edge routers will be able to send traffic destined for the other through itself.
Obviously, the whole point of a multi-homed architecture is to allow asymmetry, where a connection coming from upstream A could return through upstream B if it’s the better path. However, in my case, both of my upstreams are in the same region anyway, so the path difference is negligible right now.
My main goal was simply to ensure resilience: if an upstream BGP session goes down, there is still a path to either availability zone’s data. To achieve this, I set up dedicated WireGuard tunnels on both edges.
My network is split into two
x.x.x.192/26 goes to my home
x.x.x.0/25 and x.x.x.128/26 goes to my DC
Mikrotik (Datacenter Edge)
My Mikrotik (CCR2004) serves as an edge router in the datacenter. It has two Wireguard tunnel endpoints:
- from-vultr (to receive traffic to x.x.x.0/25 and x.x.x.128/26)
- to-home (to send traffic bound for x.x.x.192/26)
Once the wireguard connections are setup, the Mikrotik forwards traffic natively between my DC upstream BGP peer and the home tunnel.
VM Router
Similarly, the VM Router (running Ubuntu LTS with handcrafted iptables) at home has two tunnel endpoints:
- from-vultr (to receive traffic to x.x.x.192/26)
- to-dc (to send traffic bound for x.x.x.0/25 and x.x.x.128/26)
Handling Asymmetric Paths
The idea is that I am announcing my prefixes to both upstreams. Depending on which subnet inbound traffic is destined for, my edge routers (CPEs) will send the traffic to the correct site over WireGuard.
Regardless of where is the request entry point, the CPEs would be able to forward it to the right destination.
Because traffic could enter the DC but try to exit via Home (or vice versa), I had to implement policy-based routing on my VM Router to handle these asymmetric paths and prevent stateful firewall drops. This is so that if one site’s bgp session fails, the packet won’t be loss attempting to exit through a disabled path.
Through this journey, I figured out how to:
- Use fwmark and connmark to trace connection states.
- Make use of iptables mangle rules to redirect marks to a specific routing table.
- Configure these same concepts on the Mikrotik end.
The Configuration
On my Home’s VM Router, I need to create two routing tables
These routing tables (100 and 200) will be used to decide where to send the traffic to
# to dc
ip route replace default dev dc table 100
# to vultr
ip route replace default dev vultr table 200
Next, I need to tell the kernel to use these tables based on the packet marks:
# Route marked traffic to the correct table
ip rule add fwmark 100 table 100
ip rule add fwmark 200 table 200
Then I need to mark the traffic.
# These tracks all incoming packets on the dc and vultr interface and set a connmark accordingly
iptables -t mangle -A PREROUTING \
-i dc \
-m conntrack --ctstate NEW \
-m connmark --mark 0 \
-j CONNMARK --set-mark 100
iptables -t mangle -A PREROUTING \
-i vultr \
-m conntrack --ctstate NEW \
-m connmark --mark 0 \
-j CONNMARK --set-mark 200
# When a packet arrives, check if it belongs to an existing connection that has a mark. If it does, copy that connection mark onto the packet itself so the routing engine knows which routing table to use."
iptables -t mangle -A PREROUTING \
-m connmark ! --mark 0 \
-j CONNMARK --restore-mark
# If the router itself is sending a reply packet (or related traffic) for an existing connection, copy the connection mark onto the packet so it routes out the correct interface.
iptables -t mangle -A OUTPUT \
-m conntrack --ctstate ESTABLISHED,RELATED \
-j CONNMARK --restore-mark
With this, traffic coming from dc/vultr will send out the response the same way.
Similarly for the Mikrotik, we can do something like this. This is similar to the iptables configurations on the VM Routers, just using Mikrotik terminology:
# 1. Policy routing table
/routing/table
add name=to-wg fib
# 2. Keep WireGuard transport OUTSIDE policy routing
/ip/route
add dst-address=xxx/32 gateway=xxx
# 3. Internet default route for policy-routed traffic
/ip/route
add dst-address=0.0.0.0/0 gateway=vultr routing-table=to-wg
# 4. Remember connections arriving through Vultr
/ip/firewall/mangle
add chain=prerouting \
in-interface=vultr \
connection-state=new \
action=mark-connection \
new-connection-mark=from-wg \
passthrough=yes \
comment="Mark connections arriving from WG"
# 5. Send their RETURN traffic back through Vultr
/ip/firewall/mangle
add chain=prerouting \
in-interface=!vultr \
connection-state=established,related \
connection-mark=from-wg \
action=mark-routing \
new-routing-mark=to-wg \
passthrough=no \
comment="Return WG connections via WG"
And that’s it! By leveraging fwmark, connmark, and custom routing tables, both the Linux VM Router and Mikrotik CPE can now handle asymmetric routing without dropping stateful connections. If one of my upstreams goes down, traffic gracefully flows through the surviving path and across the WireGuard cross-connect.
Glad I finally got to closing this loop.