Monitoring dual WAN setup

, , Comments

Introduction

I’ve been asked how to check if both WANs are working on fail-over dual WAN linux box because from time to time when the fail-over triggers backup WAN was not working and nobody knew it.
So, the mission here is to create a way to get Zabbix checking on it and report on failure.

Implementation

To get it working I used iptables CONNMARK, multiple ip route tables, ip rules, probe scripts and users.

routing tables

I’ve created two extra routing tables: wan1 and wan2. They just have a default route to each wan.

1
2
3
# ip ro ls table wan1
default via 192.168.1.1 dev eth1
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.2
1
2
3
# ip ro ls table wan2
default via 192.168.2.1 dev eth2
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2

routing rules

Created two custom rules to redirect packets with CONNMARK 0x1 and 0x2 to table wan1 or table wan2.

1
2
3
4
5
6
7
# ip ru ls
0:      from all lookup local
220:    from all lookup 220
32764:  from all fwmark 0x2 lookup wan2
32765:  from all fwmark 0x1 lookup wan1
32766:  from all lookup main
32767:  from all lookup default

interfaces file

To get it reconfigured after reboots I’ve used post-up on interfaces file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
auto eth1
iface eth1 inet static
        address 192.168.1.2
        netmask 255.255.255.0
        broadcast 192.168.1.255
        network 192.168.1.0
        gateway 192.168.1.1

        # Prepara tabela de roteamento wan1
        post-up ip route add default via 192.168.1.1 dev eth1 table wan1
        post-up ip route add 192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.2 table wan1
        post-up ip rule add fwmark 1 lookup wan1

auto eth2                                                                                                           
iface eth2 inet static                                                                                              
        address 192.168.2.2                                                                                         
        netmask 255.255.255.0                                                                                       
        broadcast 192.168.2.255                                                                                     
        network 192.168.2.0                                                                                         
#       gateway 192.168.2.1                                                                                         

        # prepara tabela de roteamento wan2                                                                         
        post-up ip route add default via 192.168.2.1 dev eth2 table wan2                                            
        post-up ip route add 192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2 table wan2          
        post-up ip rule add fwmark 2 lookup wan2

iptables

The magic happens here. At iptables mangle table I’ve marked all packets from users testwan1 and testwan2 with their matching CONNMARK.

1
2
3
4
5
6
7
8
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

iptables -t mangle -A OUTPUT -m owner --uid-owner testwan1 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan1 -j CONNMARK --save-mark

iptables -t mangle -A OUTPUT -m owner --uid-owner testwan2 -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -m owner --uid-owner testwan2 -j CONNMARK --save-mark

Now all packets generated from user testwan1 are going to use eth1 and all packets from testwan2 are going to use eth2.

probe scripts

To get this information always available to Zabbix, two probe scripts have been created and executed by each user crontab every 5min.

1
2
3
4
5
# cat /usr/local/bin/get-eth1-public-ip
#!/bin/sh

IP=$(curl -s icanhazip.com)
echo "$IP" | tee /tmp/public-ip.wan1
1
2
3
4
5
# cat /usr/local/bin/get-eth2-public-ip
#!/bin/sh

IP=$(curl -s icanhazip.com)
echo "$IP" | tee /tmp/public-ip.wan2

cron

1
2
3
4
# crontab -u testwan1 -l

*/5 * * * *

1
2
3
# crontab -u testwan2 -l

*/5 * * * * /usr/local/bin/get-eth2-public-ip

Conclusion

Now, Zabbix can be configured to check files /tmp/public-ip.wan1 and /tmp/public-ip.wan2 and do whatever needed.
This implementation can be enhanced to don’t use crontab in favor of setuid on probe scripts and to do not generate output files.