[strongSwan] AWS EC2 IKEv2 tunnel up but no throughput
Lewis Shobbrook
l.shobbrook at base2services.com
Thu Jul 8 05:04:04 CEST 2021
Hi Noel/users,
Does no-one have any suggestions as to what we can try here?
I've looked long and hard at this and feel that I have exhausted any
obvious settings to adjust.
The only thing slightly unusual about the setup is that we are using a
single interface, but this has been documented as working so it really
should not be an issue.
Packets not traversing the tunnel is confirmed by...
swanctl --list-sas
tunnel1: #1, ESTABLISHED, IKEv2, 88d5c48c82546516_i* a2d86821f1f52625_r
local '52.8.104.97' @ 48.138.201.70[4500]
remote '68.148.15.170' @ 68.148.15.170[4500]
AES_GCM_16-256/PRF_HMAC_SHA2_384/ECP_384
established 582s ago, reauth in 85216s
tunnel1: #1, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-256
installed 582s ago, rekeying in 2218s, expires in 3018s
in c675ba46, 0 bytes, 0 packets
out 9333b51f, 0 bytes, 0 packets
local 48.138.201.64/26
remote 198.168.248.0/2
and ...
ipsec listcounters
List of IKE counters:
ikeInitRekey 0
ikeRspRekey 0
ikeChildSaRekey 0
ikeInInvalid 0
ikeInInvalidSpi 0
ikeInInitReq 0
ikeInInitRsp 2
ikeOutInitReq 2
ikeOutInitRsp 0
ikeInAuthReq 0
ikeInAuthRsp 2
ikeOutAuthReq 2
ikeOutAuthRsp 0
ikeInCrChildReq 0
ikeInCrChildRsp 0
ikeOutCrChildReq 0
ikeOutCrChildRsp 0
ikeInInfoReq 200
ikeInInfoRsp 0
ikeOutInfoReq 0
ikeOutInfoRsp 200
Any packets destined to the ip's on the rightsubnet from the left
vpn-gw itself do not escape, VPC flow logs confirm that no packets
from the vpn-gw destined for the rightsubnet traverse the vpc, and any
packets destined for the rightsubnet from the leftsubnet route to the
vpng-gw as expected but progress no further.
The problem seems to me to be tied in with the xfrm policy but this is
speculative.
Martians were detected only briefly 3 days ago with two singular
identical entries
kernel: IPv4: martian source 48.138.201.70 from 68.148.15.170, on dev eth0
All traffic between the VPN endpoints is encapsulated by NAT-T UDP 4500
tcpdump shows checksum errors for packets originating from the left
vpn-gw to the rightsubnet, but we do not see any
02:45:59.096217 IP (tos 0x0, ttl 255, id 63930, offset 0, flags
[none], proto TCP (6), length 60)
48.138.201.70.39268 > 198.168.248.4.13865: Flags [S], cksum 0xb5e1
(incorrect -> 0x6af2), seq 329764005, win 26883, options [mss
8961,sackOK,TS val 3337366083 ecr 0,nop,wscale 7], length 0
XfrmOutStateModeError count is not incremented in /proc/self/net/xfrm_stat
and we have mangle rules in place to clamp to mss to 1360.
I could not find any info out there relating to the IKE counters
I assume if we were traversing the tunnel successfully that we'd see
incremented values in ikeInInfoRsp & ikeOutInfoReq ??
also swanctl --list-sas naturally...
Hoping someone can help here.
Cheers
Lew
On Mon, 5 Jul 2021 at 19:54, Lewis Shobbrook
<l.shobbrook at base2services.com> wrote:
>
> Thanks for your reply Noel,
> Landed in my spam folder...
> I'm testing with a curl to a known endpoint from the vpn gateway
> itself and also from the associated local subnet that is accepted on
> the other side.
> Here's the output of iptables-save which has changed quite a bit over
> various efforts to realise throughput.
>
> # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021
> *mangle
> :PREROUTING ACCEPT [131559:43589022]
> :INPUT ACCEPT [129402:43459544]
> :FORWARD ACCEPT [2155:129300]
> :OUTPUT ACCEPT [131976:29447953]
> :POSTROUTING ACCEPT [134131:29577253]
> -A FORWARD -p tcp -m policy --dir in --pol ipsec -m tcp --tcp-flags
> SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360
> -A FORWARD -p tcp -m policy --dir out --pol ipsec -m tcp --tcp-flags
> SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360
> COMMIT
> # Completed on Mon Jul 5 09:52:05 2021
> # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021
> *nat
> :PREROUTING ACCEPT [0:0]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> :POSTROUTING ACCEPT [0:0]
> -A POSTROUTING -m policy --dir out --pol ipsec -j ACCEPT
> -A POSTROUTING -s 48.138.201.64/26 -o eth0 -m policy --dir out --pol
> ipsec -j ACCEPT
> COMMIT
> # Completed on Mon Jul 5 09:52:05 2021
> # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021
> *filter
> :INPUT ACCEPT [42760:14832665]
> :FORWARD ACCEPT [774:46440]
> :OUTPUT ACCEPT [43602:9751014]
> COMMIT
>
>
> Cheers,
>
> Lew
>
>
>
> Lewis Shobbrook
> Team Lead - DevOps
>
> base2Services | The Cloud Services People
> T 1300 713 559 E l.shobbrook at base2services.com
> Lvl 21, 303 Collins St, Melbourne VIC 3000
> base2services.com.au
>
>
>
>
> On Mon, 5 Jul 2021 at 18:19,
> <noel.kuntze+strongswan-users-ml at thermi.consulting> wrote:
> >
> > Hello Lew,
> >
> > How exactly are you testing the tunnel?
> > Also, please provide the output of iptables-save.
> >
> > Kind regards
> > Noel
> >
> > Am July 5, 2021 7:28:19 AM UTC schrieb Lewis Shobbrook <l.shobbrook at base2services.com>:
> >>
> >> Hi Guys,
> >> I have an IKEv2 tunnel that is established and up, but I am unable to
> >> route any packets across it.
> >> All ACL's are configured to allow UDP 500,4500 & protocols 50, 51 &
> >> icmp to/from the non aws end.
> >> Local iptables are permissive with default policys ACCEPT
> >> Security groups also allow anything outbound and the above ports &
> >> protos inbound.
> >> Here are a few particulars typically requested ahead of time.
> >> ip_forward is enabled rp_filter disabled as follows...
> >> net.ipv4.ip_forward = 1
> >> net.ipv4.conf.all.send_redirects = 0
> >> net.ipv4.conf.default.send_redirects = 0
> >> net.ipv4.tcp_max_syn_backlog = 1280
> >> net.ipv4.icmp_echo_ignore_broadcasts = 1
> >> net.ipv4.conf.all.accept_source_route = 0
> >> net.ipv4.conf.all.accept_redirects = 0
> >> net.ipv4.conf.all.secure_redirects = 0
> >> net.ipv4.conf.all.log_martians = 1
> >> net.ipv4.conf.default.accept_source_route = 0
> >> net.ipv4.conf.default.accept_redirects = 0
> >> net.ipv4.conf.default.secure_redirects = 0
> >> net.ipv4.icmp_echo_ignore_broadcasts = 1
> >> net.ipv4.icmp_ignore_bogus_error_responses = 1
> >> net.ipv4.tcp_syncookies = 1
> >> net.ipv4.conf.all.rp_filter = 0
> >> net.ipv4.conf.default.rp_filter = 0
> >> net.ipv4.tcp_mtu_probing = 1
> >>
> >> tcpdump just shows one way requests
> >> Looking at the 220 rt table I can see that the auto added route
> >> appears to be correct, and the xfrm policy nothing obvious to me, with
> >> no xfrm vi's used.
> >> Obfuscated ip's naturally...
> >> ip r li ta 220
> >> 198.168.248.0/29 via 48.138.201.65 dev eth0 proto static src 48.138.201.70
> >> ip xfrm policy
> >> src 48.138.201.64/26 dst 198.168.248.0/29
> >> dir out priority 371839 ptype main
> >> tmpl src 48.138.201.70 dst 68.169.15.170
> >> proto esp spi 0x2c1e849e reqid 1 mode tunnel
> >> src 198.168.248.0/29 dst 48.138.201.64/26
> >> dir fwd priority 371839 ptype main
> >> tmpl src 68.148.15.170 dst 48.138.201.70
> >> proto esp reqid 1 mode tunnel
> >> src 198.168.248.0/29 dst 48.138.201.64/26
> >> dir in priority 371839 ptype main
> >> tmpl src 68.148.15.170 dst 48.138.201.70
> >> proto esp reqid 1 mode tunnel
> >> src 0.0.0.0/0 dst 0.0.0.0/0
> >> socket in priority 0 ptype main
> >> src 0.0.0.0/0 dst 0.0.0.0/0
> >> socket out priority 0 ptype main
> >> src 0.0.0.0/0 dst 0.0.0.0/0
> >> socket in priority 0 ptype main
> >> src 0.0.0.0/0 dst 0.0.0.0/0
> >> socket out priority 0 ptype main
> >> src ::/0 dst ::/0
> >> socket in priority 0 ptype main
> >> src ::/0 dst ::/0
> >> socket out priority 0 ptype main
> >> src ::/0 dst ::/0
> >> socket in priority 0 ptype main
> >> src ::/0 dst ::/0
> >> socket out priority 0 ptype main
> >> src 198.168.248.0/29 dst 48.138.201.64/26
> >> dir fwd priority 371840 ptype main
> >> tmpl src 68.148.15.170 dst 48.138.201.70
> >> proto esp reqid 2 mode tunnel
> >> src 198.168.248.0/29 dst 48.138.201.64/26
> >> dir in priority 371840 ptype main
> >> tmpl src 68.148.15.170 dst 48.138.201.70
> >> proto esp reqid 2 mode tunnel
> >> src 48.138.201.64/26 dst 198.168.248.0/29
> >> dir out priority 371840 ptype main
> >> tmpl src 48.138.201.70 dst 68.169.15.170
> >> proto esp reqid 2 mode tunnel
> >>
> >> ipsec statusall
> >> Status of IKE charon daemon (strongSwan 5.7.2, Linux
> >> 4.14.232-177.418.amzn2.x86_64, x86_64):
> >> uptime: 54 minutes, since Jul 05 06:10:30 2021
> >> malloc: sbrk 2846720, mmap 0, used 1023696, free 1823024
> >> worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0,
> >> scheduled: 1
> >> loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4
> >> md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1
> >> pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp
> >> curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink
> >> resolve socket-default farp stroke vici updown eap-identity eap-sim
> >> eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2
> >> eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic
> >> xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
> >> Listening IP addresses:
> >> 48.138.201.70
> >> Connections:
> >> tunnel1: %any...68.148.15.170 IKEv2
> >> tunnel1: local: uses pre-shared key authentication
> >> tunnel1: remote: uses pre-shared key authentication
> >> tunnel1: child: 48.138.201.64/26 === 198.168.248.0/29 TUNNEL
> >> Security Associations (1 up, 0 connecting):
> >> tunnel1[2]: ESTABLISHED 24 minutes ago,
> >> 48.138.201.70[48.138.201.70]...65.169.15.170[65.169.15.170]
> >> tunnel1[2]: IKEv2 SPIs: d3e732eb14d78aec_i* b06860d1ceee2f9a_r,
> >> rekeying disabled
> >> tunnel1[2]: IKE proposal: AES_GCM_16_256/PRF_HMAC_SHA2_384/ECP_384
> >> tunnel1{2}: INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: cb651f89_i 479cff91_o
> >> tunnel1{2}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying disabled
> >> tunnel1{2}: 48.138.201.64/26 === 198.168.248.0/30
> >>
> >> VPC flow logs show no proto 50, only 4500 & 500.
> >> I've also tried to clamp mss not that I expect it would have changed 0
> >> throughput
> >> iptables -t mangle -A FORWARD -m policy --pol ipsec --dir in -p tcp -m
> >> tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS
> >> --set-mss 1360
> >>
> >> I've spent hours searching but have not found anything to help.
> >> Hoping someone here may have a suggestion ot two?
> >> Cheers,
> >>
> >> Lew
> >
> >
> > Sent from mobile
More information about the Users
mailing list