[strongSwan] GRE over IPSec dual connections from road warriors strange behavior
John Serink
john_serink at trimble.com
Thu Mar 31 17:14:33 CEST 2022
Hello:
I have 2 questions:
1. If the debug level is changed to 2 such as:
config setup
charondebug="cfg 2, dmn 2, ike 2, net 2"
Does it have much operational affect on strongswan running on a roadwarrior in terms of CPU
loading?
2. Strange problem with GRE tunnels.
I have strongswan 5.8.4 running on a Teltonika RUT-950. We have hundreds of
these units running in India and I am remotely manageing them from Singapore.
I have a RUT-950 router with me in Singapore that I use for testing.
We have two Cisco 4431 routers called CC and CC2 in India on static IPs. My problem is
happening on the RUT-950 units that are connecting to both Ciscos using different IPSec
connection profiles.
We are doing it this way as this makes the RUT-950 a drop in replacement for the
Digi WR seriers of routers.
In my description, router CC(tunnel SOI) will be IP a.b.c.d and router CC2(Tunnel SOIMP) will
be c.d.e.f.
Here is the /etc/ipsec.conf file from the RUT-950:
root at CORS525:/# cat /etc/ipsec.conf
# generated by /etc/init.d/ipsec
conn %default
rekeymargin=9m
rekeyfuzz=100%
mobike=no
config setup
charondebug="cfg 2, dmn 2, ike 2, net 2"
conn SOICC
leftid=keyid:CORS525
leftauth=psk
rightauth=psk
leftsubnet=2.2.4.13/32[gre]
right=a.b.c.d
rightid=keyid:CCrouter
keyexchange=ikev2
authby=secret
leftfirewall=yes
rightfirewall=no
auto=start
type=tunnel
aggressive=no
dpdaction=restart
dpddelay=30
dpdtimeout=30
forceencaps=no
keyingtries=%forever
ike=aes256-sha256-modp2048
ikelifetime=5h
esp=aes256-sha256-modp2048
keylife=4h
closeaction=restart
rightsubnet=1.1.1.10/32[gre]
conn SOICCMP
leftid=keyid:CORS525
leftauth=psk
rightauth=psk
leftsubnet=3.3.4.13/32[gre]
right=c.d.e.f
rightid=keyid:CC2router
keyexchange=ikev2
authby=secret
leftfirewall=yes
rightfirewall=no
auto=start
type=tunnel
aggressive=no
dpdaction=restart
dpddelay=30
dpdtimeout=30
forceencaps=no
keyingtries=%forever
ike=aes256-sha256-modp2048
ikelifetime=5h
esp=aes256-sha256-modp2048
keylife=4h
closeaction=restart
rightsubnet=1.1.1.12/32[gre]
The GRE tunnels setup is done from the /etc/rc.local file which is run at bootup:
root at CORS525:/# cat /etc/rc.local
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
# Issue an ipsec stop.
/usr/bin/logger -t rc.local "STARTING of the RC.LOCAL file"
echo 1 > /proc/sys/net/ipv4/conf/default/accept_local
echo 1 > /proc/sys/net/ipv4/conf/all/accept_local
#/etc/init.d/ipsec disable
/etc/init.d/ipsec stop
ip addr del dev SOI 192.168.198.49/30
ip link set dev SOI down
ip tun del SOI
ip addr del dev tap0 2.2.4.13/32
ip tuntap del tap0 mode tap
sleep 1
ip addr del dev SOIMP 172.16.165.49/30
ip link set dev SOIMP down
ip tun del SOIMP
ip addr del dev tap1 3.3.4.13/32
ip tuntap del tap1 mode tap
sleep 1
ip tuntap add name tap0 mode tap
ip addr flush dev tap0
ip addr add 2.2.4.13/32 brd + dev tap0
ip link set dev tap0 up
sleep 1
ip tuntap add name tap1 mode tap
ip addr flush dev tap1
ip addr add 3.3.4.13/32 brd + dev tap1
ip link set dev tap1 up
sleep 1
ip tunnel add SOI mode gre remote 1.1.1.10 local 2.2.4.13 ttl 255
ip link set SOI mtu 1300
ip link set SOI up
ip addr add 192.168.198.49/30 peer 192.168.198.50/30 brd + dev SOI
sleep 1
ip tunnel add SOIMP mode gre remote 1.1.1.12 local 3.3.4.13 ttl 255
ip link set SOIMP mtu 1300
ip link set SOIMP up
ip addr add 172.16.165.49/30 peer 172.16.165.50/30 brd + dev SOIMP
sleep 1
ip route add 172.16.0.0/16 dev SOIMP
ip route add 192.168.0.0/16 dev SOI
#sh /root/isalive0.14.sh 192.168.48.1 172.16.48.1 &
/usr/bin/logger -t rc.local "End of the RC.LOCAL file"
sh /root/startipsec.sh &
exit 0
The startipsec.sh is just a file that delays the /etc/init.d/ipsec start by 30 seconds to
allow the GPRS or fiber interfaces to come up.
So my GRE tunnels are SOI to network 192.168.0.0/16 located on router CC and SOIMP to network
172.16.0.0/16 located on router CC2.
Cisco GRE tunnel definition from router CC:
CCrouter#sh run int tun525
Building configuration...
Current configuration : 255 bytes
!
interface Tunnel525
description connection to CORS350
ip address 192.168.198.50 255.255.255.252
no ip redirects
ip mtu 1300
ip tcp adjust-mss 1260
keepalive 15 3
tunnel source 1.1.1.10
tunnel destination 2.2.4.13
tunnel path-mtu-discovery
end
GRE tunnel definition from router CC2:
CC2router#sh run int tun525
Building configuration...
Current configuration : 254 bytes
!
interface Tunnel525
description connection to CORS525
ip address 172.16.165.50 255.255.255.252
no ip redirects
ip mtu 1300
ip tcp adjust-mss 1260
keepalive 15 3
tunnel source 1.1.1.12
tunnel destination 3.3.4.13
tunnel path-mtu-discovery
end
Here are the issues:
1. From my test router here in Singapore, this setup works perfectly everytime, all the time.
I can leave it running over night, reboot the router, it always comes up. Works fine.
2. In India it works correctly about 50% of the time. The main issue is the IPSec tunnels come
up fine but the GRE tunnels cannot send data. Either one or both GRE tunnels cannot send data.
Sometimes an "ipsec down SOICC" followed by an "ipsec up SOICC" will fix it, many times not.
I have looked at the logs and they list only information about the IPSec tunnels(which
typically work) and nothing about the GRE.
The interface dumps look the same whether the GRE tunnels are working or not:
ip link:
root at CORS525:/# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN
mode DEFAULT group default qlen 1000
link/ether 00:1e:42:32:a3:c0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group
default qlen 1000
link/ether 00:1e:42:32:a3:c1 brd ff:ff:ff:ff:ff:ff
4: ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default
link/tunnel6 :: brd ::
5: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
link/ether 9e:cd:76:2a:e8:6e brd ff:ff:ff:ff:ff:ff
6: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
link/ether 96:21:6c:22:a2:1d brd ff:ff:ff:ff:ff:ff
7: gre0 at NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default
link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group
default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN mode DEFAULT group default
link/gre6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
11: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode
DEFAULT group default qlen 1000
link/ether 0e:d9:66:d1:ca:d6 brd ff:ff:ff:ff:ff:ff
12: br-lan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT
group default
link/ether 00:1e:42:32:a3:c0 brd ff:ff:ff:ff:ff:ff
15: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN
mode DEFAULT group default qlen 500
link/ether 5e:68:84:28:8e:98 brd ff:ff:ff:ff:ff:ff
16: tap1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT
group default qlen 500
link/ether 02:c2:df:2e:f6:77 brd ff:ff:ff:ff:ff:ff
17: SOI at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN mode
DEFAULT group default
link/gre 2.2.4.13 peer 1.1.1.10
18: SOIMP at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN mode
DEFAULT group default
link/gre 3.3.4.13 peer 1.1.1.12
ip addr:
root at CORS525:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN
group default qlen 1000
link/ether 00:1e:42:32:a3:c0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen
1000
link/ether 00:1e:42:32:a3:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::21e:42ff:fe32:a3c1/64 scope link
valid_lft forever preferred_lft forever
4: ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default
link/tunnel6 :: brd ::
5: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether 9e:cd:76:2a:e8:6e brd ff:ff:ff:ff:ff:ff
6: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether 96:21:6c:22:a2:1d brd ff:ff:ff:ff:ff:ff
7: gre0 at NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default
link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default
link/gre6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
11: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group
default qlen 1000
link/ether 0e:d9:66:d1:ca:d6 brd ff:ff:ff:ff:ff:ff
inet 100.77.158.227/32 brd 100.77.158.227 scope global wwan0
valid_lft forever preferred_lft forever
inet6 fe80::cd9:66ff:fed1:cad6/64 scope link
valid_lft forever preferred_lft forever
12: br-lan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group
default
link/ether 00:1e:42:32:a3:c0 brd ff:ff:ff:ff:ff:ff
inet 192.168.67.97/29 brd 192.168.67.103 scope global br-lan
valid_lft forever preferred_lft forever
15: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN
group default qlen 500
link/ether 5e:68:84:28:8e:98 brd ff:ff:ff:ff:ff:ff
inet 2.2.4.13/32 scope global tap0
valid_lft forever preferred_lft forever
16: tap1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default
qlen 500
link/ether 02:c2:df:2e:f6:77 brd ff:ff:ff:ff:ff:ff
inet 3.3.4.13/32 scope global tap1
valid_lft forever preferred_lft forever
17: SOI at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN group
default
link/gre 2.2.4.13 peer 1.1.1.10
inet 192.168.198.49 peer 192.168.198.50/30 brd 192.168.198.51 scope global SOI
valid_lft forever preferred_lft forever
18: SOIMP at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN group
default
link/gre 3.3.4.13 peer 1.1.1.12
inet 172.16.165.49 peer 172.16.165.50/30 brd 172.16.165.51 scope global SOIMP
valid_lft forever preferred_lft forever
ip route:
root at CORS525:/# ip route
default dev wwan0 scope link
default via 192.168.1.2 dev eth1 metric 10
100.77.158.227 dev wwan0 proto static scope link metric 10
172.16.0.0/16 dev SOIMP scope link
172.16.165.48/30 dev SOIMP proto kernel scope link src 172.16.165.49
192.168.0.0/16 dev SOI scope link
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.1
192.168.67.96/29 dev br-lan proto kernel scope link src 192.168.67.97
192.168.198.48/30 dev SOI proto kernel scope link src 192.168.198.49
root at CORS525:/# ip xfrm policy
src 3.3.4.13/32 dst 1.1.1.12/32 proto gre
dir out priority 366975
tmpl src 100.77.158.227 dst 164.100.196.79
proto esp spi 0xe1723ece reqid 2 mode tunnel
src 1.1.1.12/32 dst 3.3.4.13/32 proto gre
dir fwd priority 366975
tmpl src 164.100.196.79 dst 100.77.158.227
proto esp reqid 2 mode tunnel
src 1.1.1.12/32 dst 3.3.4.13/32 proto gre
dir in priority 366975
tmpl src 164.100.196.79 dst 100.77.158.227
proto esp reqid 2 mode tunnel
src 2.2.4.13/32 dst 1.1.1.10/32 proto gre
dir out priority 366975
tmpl src 100.77.158.227 dst 103.205.244.106
proto esp spi 0x32bb69c5 reqid 1 mode tunnel
src 1.1.1.10/32 dst 2.2.4.13/32 proto gre
dir fwd priority 366975
tmpl src 103.205.244.106 dst 100.77.158.227
proto esp reqid 1 mode tunnel
src 1.1.1.10/32 dst 2.2.4.13/32 proto gre
dir in priority 366975
tmpl src 103.205.244.106 dst 100.77.158.227
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
If I set the unit to use only one of the connection profiles by using auto=add or auto=ignore
the one tunnel will come up and work properly. If I set both to come up these are my results:
Both work ~50% of the time:
IPSec on both up, one GRE works, the other does not, ~20%,
IPSec on one up, GRE works on that one, IPSec will not come up on the other. ~20%.
IPSec will not come up on either. ~10%
Note that our physical links are not very good. We have unreliable LTE and unreliable BB
fiber.
The debug logs from the Cisco end show that the tunnels come up and are torn down by the
strongswan end. Many of the failures are caused by dropped packets to be sure.
As a sanity check, does what I am doing even make sense?
Any tips on why I would get this strange performance anomalies over and above the dropped
packets?
The dropped packets all happen at the RUT-950 end so they should affect both links equally but
as I have described, they don't so I'm thinking I may be making a fundamental mistake
somewhere.
Cheers,
John
--
John Edward Serink
Product Applications Engineer,
Advanced Positioning
Trimble Navigation Singapore PTE Ltd.
3 Harbourfront Place,
#13-02 Harbourfront Tower Two,
Co. Reg. No. 199204958W
Singapore 099254
Tel 65-6871-5878
Fax 65-6871-5879
DID 65-6871-5873
HP 65-9129-4250
Skype: johnserink
More information about the Users
mailing list