[strongSwan] Issues with HA configuration

[Whisker, Peter]

Hi

I'm struggling with the HA cluster configuration on Centos 7 (I have what seems to work well running on Debian Jessie). I have now got to a stage where the IPSec side of the HA seems to be working (one node is passive and one is active) but I'm having issues routing to the protected network behind the Strongwan cluster.

I have patched Linux 3.10 (on Centos 7) using the 3.11 CLUSTERIP/XFRM patches in ha-3.11-abicompat.patch.bz2 . One needed to be done manually as there was a minor variable name change between 3.10 and 3.11.

I have set up the two CLUSTERIP rules on the two boxes as:

[root at irisp-gsgw-1a ipv4]# cat /sbin/ifup-local
#!/bin/sh

if [ "$1" = "ens224" ]; then
echo "Setting up HA cluster (Air side) address on $1"
/usr/sbin/ip address add 10.0.0.2/24 dev ens224
/usr/sbin/iptables -A INPUT -i ens224 -d 10.0.0.2 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:64:20    --total-nodes 2 --local-node 0
fi

if [ "$1" == "ens256" ]; then
echo "Setting up HA cluster (Ground side) address on $1"
/usr/sbin/ip address add 10.10.0.200/24 dev ens256
/usr/sbin/iptables -A INPUT -i ens256 -d 10.10.0.200 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:65:40    --total-nodes 2 --local-node 0
fi

I have enabled forwarding and there are no iptables rules to stop it:

[root at irisp-gsgw-1a ipv4]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward=1

[root at irisp-gsgw-1a ipv4]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
CLUSTERIP  all  --  anywhere             irisp-gsgw-1a        CLUSTERIP hashmode=sourceip clustermac=01:00:5E:00:64:20 total_nodes=2 local_node=0 hash_init=0
CLUSTERIP  all  --  anywhere             irisp-gsgw-1a        CLUSTERIP hashmode=sourceip clustermac=01:00:5E:00:65:40 total_nodes=2 local_node=0 hash_init=0
LOGGING    all  --  anywhere             anywhere
LOG        all  --  anywhere             anywhere             limit: avg 15/min burst 5 LOG level debug prefix "IPTABLES DENIED[INPUT]:  "

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain LOGGING (1 references)
target     prot opt source               destination
[root at irisp-gsgw-1a ipv4]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

The network addresses are there:
[root at irisp-gsgw-1a ipv4]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens161: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7b:03:c9 brd ff:ff:ff:ff:ff:ff
    inet 1xx.xxx.89.92/22 brd 158.234.91.255 scope global dynamic ens161
       valid_lft 8323sec preferred_lft 8323sec
    inet6 fe80::20c:29ff:fe7b:3c9/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7b:03:ab brd ff:ff:ff:ff:ff:ff
    inet 172.31.255.201/24 brd 172.31.255.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe7b:3ab/64 scope link
       valid_lft forever preferred_lft forever
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7b:03:b5 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/24 brd 10.0.0.255 scope global ens224
       valid_lft forever preferred_lft forever
    inet 10.0.0.2/24 scope global secondary ens224
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe7b:3b5/64 scope link
       valid_lft forever preferred_lft forever
5: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7b:03:bf brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.201/24 brd 10.10.0.255 scope global ens256
       valid_lft forever preferred_lft forever
    inet 10.10.0.200/24 scope global secondary ens256
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe7b:3bf/64 scope link
       valid_lft forever preferred_lft forever
[root at irisp-gsgw-1a ipv4]#

ens192 is the hearbeat, ens224 is the encrypted link over the public network and ens256 is the link to the protected network.

I have a test server on 10.10.0.203 which has a route to 10.10.0.200 (the VIP) for the remote client which is on 172.26.0.1

[root at irisp-gsgw-1a ipv4]# ipsec status
Security Associations (1 up, 0 connecting):
        asgw[1]: ESTABLISHED 3 hours ago, 10.0.0.2[C=GB, O=Iris Service Provider, OU=airline device sponsor, CN=GSGW28Aug15]...10.126.0.2[ICAO76543210.IMSI543210987654321.CRD123456789abc.00]
        asgw{1}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c86c1ca9_i f5d3d5f9_o
        asgw{1}:   10.10.0.0/24 === 172.26.0.1/32

When I ping 172.26.0.1 from 10.10.0.203 I can see the ICMP requests appearing on both StrongSwan nodes in tcpdump but neither seem to deal with it (ie it gets dropped).
When I ping 10.10.0.203 from 172.26.0.1, the ICMP request gets put onto the 10.10.0.0/24 network and 10.10.0.203 replies however as before with the ICMP request the reply gets dropped.

I believe from dropwatch that the reply is being dropped in ip_forward+1b8 which is the code at the end:

drop:
        kfree_skb(skb);
        return NET_RX_DROP;
}

One question I have is about the 
iptables -A INPUT -i ens224 -d 10.0.0.2 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:64:20    --total-nodes 2 --local-node 0
iptables -A INPUT -i ens256 -d 10.10.0.200 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:65:40    --total-nodes 2 --local-node 0

lines looking similar like this (which I have send elsewhere). If the hash is on SOURCE IP then won't it potentially hash to a different segment depending on the direction of the message? Or am I missing something? I would really appreciate some guidance! Could there be one of the settings (perhaps related to multicast ARP or martians or something which I might also need to change?)

Many thanks
Peter