<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m struggling with the HA cluster configuration on Centos 7 (I have what seems to work well running on Debian Jessie). I have now got to a stage where the IPSec side of the HA seems to be working (one node is passive and one is active)
but I’m having issues routing to the protected network behind the Strongwan cluster.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have patched Linux 3.10 (on Centos 7) using the 3.11 CLUSTERIP/XFRM patches in ha-3.11-abicompat.patch.bz2 . One needed to be done manually as there was a minor variable name change between 3.10 and 3.11.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have set up the two CLUSTERIP rules on the two boxes as:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# cat /sbin/ifup-local<o:p></o:p></p>
<p class="MsoNormal">#!/bin/sh<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">if [ "$1" = "ens224" ]; then<o:p></o:p></p>
<p class="MsoNormal">echo "Setting up HA cluster (Air side) address on $1"<o:p></o:p></p>
<p class="MsoNormal">/usr/sbin/ip address add 10.0.0.2/24 dev ens224<o:p></o:p></p>
<p class="MsoNormal">/usr/sbin/iptables -A INPUT -i ens224 -d 10.0.0.2 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:64:20 --total-nodes 2 --local-node 0<o:p></o:p></p>
<p class="MsoNormal">fi<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">if [ "$1" == "ens256" ]; then<o:p></o:p></p>
<p class="MsoNormal">echo "Setting up HA cluster (Ground side) address on $1"<o:p></o:p></p>
<p class="MsoNormal">/usr/sbin/ip address add 10.10.0.200/24 dev ens256<o:p></o:p></p>
<p class="MsoNormal">/usr/sbin/iptables -A INPUT -i ens256 -d 10.10.0.200 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:65:40 --total-nodes 2 --local-node 0<o:p></o:p></p>
<p class="MsoNormal">fi<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have enabled forwarding and there are no iptables rules to stop it:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# cat /etc/sysctl.conf<o:p></o:p></p>
<p class="MsoNormal"># System default settings live in /usr/lib/sysctl.d/00-system.conf.<o:p></o:p></p>
<p class="MsoNormal"># To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file<o:p></o:p></p>
<p class="MsoNormal">#<o:p></o:p></p>
<p class="MsoNormal"># For more information, see sysctl.conf(5) and sysctl.d(5).<o:p></o:p></p>
<p class="MsoNormal">net.ipv4.ip_forward=1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# iptables -L<o:p></o:p></p>
<p class="MsoNormal">Chain INPUT (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal">CLUSTERIP all -- anywhere irisp-gsgw-1a CLUSTERIP hashmode=sourceip clustermac=01:00:5E:00:64:20 total_nodes=2 local_node=0 hash_init=0<o:p></o:p></p>
<p class="MsoNormal">CLUSTERIP all -- anywhere irisp-gsgw-1a CLUSTERIP hashmode=sourceip clustermac=01:00:5E:00:65:40 total_nodes=2 local_node=0 hash_init=0<o:p></o:p></p>
<p class="MsoNormal">LOGGING all -- anywhere anywhere<o:p></o:p></p>
<p class="MsoNormal">LOG all -- anywhere anywhere limit: avg 15/min burst 5 LOG level debug prefix "IPTABLES DENIED[INPUT]: "<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain FORWARD (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain OUTPUT (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain LOGGING (1 references)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# iptables -L -t nat<o:p></o:p></p>
<p class="MsoNormal">Chain PREROUTING (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain INPUT (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain OUTPUT (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chain POSTROUTING (policy ACCEPT)<o:p></o:p></p>
<p class="MsoNormal">target prot opt source destination<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The network addresses are there:<o:p></o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# ip addr<o:p></o:p></p>
<p class="MsoNormal">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN<o:p></o:p></p>
<p class="MsoNormal"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<o:p></o:p></p>
<p class="MsoNormal"> inet 127.0.0.1/8 scope host lo<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet6 ::1/128 scope host<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal">2: ens161: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether 00:0c:29:7b:03:c9 brd ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> inet 1xx.xxx.89.92/22 brd 158.234.91.255 scope global dynamic ens161<o:p></o:p></p>
<p class="MsoNormal"> valid_lft 8323sec preferred_lft 8323sec<o:p></o:p></p>
<p class="MsoNormal"> inet6 fe80::20c:29ff:fe7b:3c9/64 scope link<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal">3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether 00:0c:29:7b:03:ab brd ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> inet 172.31.255.201/24 brd 172.31.255.255 scope global ens192<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet6 fe80::20c:29ff:fe7b:3ab/64 scope link<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal">4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether 00:0c:29:7b:03:b5 brd ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> inet 10.0.0.11/24 brd 10.0.0.255 scope global ens224<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet 10.0.0.2/24 scope global secondary ens224<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet6 fe80::20c:29ff:fe7b:3b5/64 scope link<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal">5: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether 00:0c:29:7b:03:bf brd ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> inet 10.10.0.201/24 brd 10.10.0.255 scope global ens256<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet 10.10.0.200/24 scope global secondary ens256<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal"> inet6 fe80::20c:29ff:fe7b:3bf/64 scope link<o:p></o:p></p>
<p class="MsoNormal"> valid_lft forever preferred_lft forever<o:p></o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">ens192 is the hearbeat, ens224 is the encrypted link over the public network and ens256 is the link to the protected network.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a test server on 10.10.0.203 which has a route to 10.10.0.200 (the VIP) for the remote client which is on 172.26.0.1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@irisp-gsgw-1a ipv4]# ipsec status<o:p></o:p></p>
<p class="MsoNormal">Security Associations (1 up, 0 connecting):<o:p></o:p></p>
<p class="MsoNormal"> asgw[1]: ESTABLISHED 3 hours ago, 10.0.0.2[C=GB, O=Iris Service Provider, OU=airline device sponsor, CN=GSGW28Aug15]...10.126.0.2[ICAO76543210.IMSI543210987654321.CRD123456789abc.00]<o:p></o:p></p>
<p class="MsoNormal"> asgw{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c86c1ca9_i f5d3d5f9_o<o:p></o:p></p>
<p class="MsoNormal"> asgw{1}: 10.10.0.0/24 === 172.26.0.1/32<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I ping 172.26.0.1 from 10.10.0.203 I can see the ICMP requests appearing on both StrongSwan nodes in tcpdump but neither seem to deal with it (ie it gets dropped).<o:p></o:p></p>
<p class="MsoNormal">When I ping 10.10.0.203 from 172.26.0.1, the ICMP request gets put onto the 10.10.0.0/24 network and 10.10.0.203 replies however as before with the ICMP request the reply gets dropped.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I believe from dropwatch that the reply is being dropped in ip_forward+1b8 which is the code at the end:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">drop:<o:p></o:p></p>
<p class="MsoNormal"> kfree_skb(skb);<o:p></o:p></p>
<p class="MsoNormal"> return NET_RX_DROP;<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One question I have is about the <o:p></o:p></p>
<p class="MsoNormal">iptables -A INPUT -i ens224 -d 10.0.0.2 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:64:20 --total-nodes 2 --local-node 0<o:p></o:p></p>
<p class="MsoNormal">iptables -A INPUT -i ens256 -d 10.10.0.200 -j CLUSTERIP --new --hashmode sourceip --clustermac 01:00:5e:00:65:40 --total-nodes 2 --local-node 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">lines looking similar like this (which I have send elsewhere). If the hash is on SOURCE IP then won’t it potentially hash to a different segment depending on the direction of the message? Or am I missing something? I would really appreciate
some guidance! Could there be one of the settings (perhaps related to multicast ARP or martians or something which I might also need to change?)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Many thanks<o:p></o:p></p>
<p class="MsoNormal">Peter<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>