Monitoring dual WAN setup

Introduction

I’ve been asked how to check if both WANs are working on fail-over dual WAN linux box because from time to time when the fail-over triggers backup WAN was not working and nobody knew it.
So, the mission here is to create a way to get Zabbix checking on it and report on failure.

Implementation

To get it working I used iptables CONNMARK, multiple ip route tables, ip rules, probe scripts and users.

routing tables

I’ve created two extra routing tables: wan1 and wan2. They just have a default route to each wan.

1
2
3
# ip ro ls table wan1
default via 192.168.1.1 dev eth1
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.2
1
2
3
# ip ro ls table wan2
default via 192.168.2.1 dev eth2
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.2

routing rules

Created two custom rules to redirect packets with CONNMARK 0x1 and 0x2 to table wan1 or table wan2.

1
2
3
4
5
6
7
# ip ru ls
0: from all lookup local
220: from all lookup 220
32764: from all fwmark 0x2 lookup wan2
32765: from all fwmark 0x1 lookup wan1
32766: from all lookup main
32767: from all lookup default

interfaces file

To get it reconfigured after reboots I’ve used post-up on interfaces file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
auto eth1
iface eth1 inet static
address 192.168.1.2
netmask 255.255.255.0
broadcast 192.168.1.255
network 192.168.1.0
gateway 192.168.1.1
# Prepara tabela de roteamento wan1
post-up ip route add default via 192.168.1.1 dev eth1 table wan1
post-up ip route add 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.2 table wan1
post-up ip rule add fwmark 1 lookup wan1
auto eth2
iface eth2 inet static
address 192.168.2.2
netmask 255.255.255.0
broadcast 192.168.2.255
network 192.168.2.0
# gateway 192.168.2.1
# prepara tabela de roteamento wan2
post-up ip route add default via 192.168.2.1 dev eth2 table wan2
post-up ip route add 192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.2 table wan2
post-up ip rule add fwmark 2 lookup wan2

iptables

The magic happens here. At iptables mangle table I’ve marked all packets from users testwan1 and testwan2 with their matching CONNMARK.

1
2
3
4
5
6
7
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan1 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan1 -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan2 -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan2 -j CONNMARK --save-mark

Now all packets generated from user testwan1 are going to use eth1 and all packets from testwan2 are going to use eth2.

probe scripts

To get this information always available to Zabbix, two probe scripts have been created and executed by each user crontab every 5min.

1
2
3
4
5
# cat /usr/local/bin/get-eth1-public-ip
#!/bin/sh
IP=$(curl -s icanhazip.com)
echo "$IP" | tee /tmp/public-ip.wan1
1
2
3
4
5
# cat /usr/local/bin/get-eth2-public-ip
#!/bin/sh
IP=$(curl -s icanhazip.com)
echo "$IP" | tee /tmp/public-ip.wan2

cron

1
2
3
# crontab -u testwan1 -l
*/5 * * * *
1
2
3
# crontab -u testwan2 -l
*/5 * * * * /usr/local/bin/get-eth2-public-ip

Conclusion

Now, Zabbix can be configured to check files /tmp/public-ip.wan1 and /tmp/public-ip.wan2 and do whatever needed.
This implementation can be enhanced to don’t use crontab in favor of setuid on probe scripts and to do not generate output files.