[strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
Noel Kuntze
noel.kuntze at thermi.consulting
Fri Sep 3 21:44:26 CEST 2021
https://www.strongswan.org/testing/testresults/route-based/net2net-vti/index.html
Am 03.09.21 um 21:17 schrieb Tiago Stoco:
> Hi Noel,
>
> Let me add a bit of context before presenting the changes and its implications.
>
> [root at arch-linux ~]# ip -c addr
> 2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
> link/ether aa:e4:30:3e:84:dbbrd ff:ff:ff:ff:ff:ff
> altname enp0s18
> inet 192.168.45.30/24 scope global ens18
> valid_lft forever preferred_lft forever
> inet6 fe80::a8e4:30ff:fe3e:84db/64 scope link
> valid_lft forever preferred_lft forever
> 7: ip_vti1 at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN group default qlen 1000
> link/ipip 192.168.45.30peer 192.168.45.10
> inet 10.10.10.2peer 10.10.10.1/32 scope global ip_vti1
> valid_lft forever preferred_lft forever
> inet6 fe80::5efe:c0a8:2d1e/64 scope link
> valid_lft forever preferred_lft forever
>
>
> *ens18 - *is the WAN interface.
> *ip_vti1 - *is the VTI interface.
>
>
> Before enabling XFRM as you have proposed for |ens18| it was as follows 👇
>
> net.ipv4.conf.ens18.rp_filter = 2
> net.ipv4.conf.ens18.disable_policy = 1
> net.ipv4.conf.ens18.disable_xfrm = 0
>
> I am changing |ens18| and |ip_vti1| as follows 👇
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ip_vti1
> net.ipv4.conf.ip_vti1.disable_policy = 1
> net.ipv4.conf.ip_vti1.rp_filter = 0
> net.ipv4.conf.ip_vti1.disable_xfrm = 1
> net.ipv4.ip_forward = 1
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ens18
> net.ipv4.conf.ens18.disable_policy = 1
> net.ipv4.conf.ens18.rp_filter = 2
> net.ipv4.conf.ens18.disable_xfrm = 0
> net.ipv4.ip_forward = 1
>
> Also, I have changed the ikey and okey for the VTI to create a distinction between traffic ingressing and egressing the interface.
>
> [root at arch-linux ~]# ip -d link show dev ip_vti1
> 7: ip_vti1 at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
> link/ipip 192.168.45.30 peer 192.168.45.10 promiscuity 0 minmtu 0 maxmtu 0
> vti remote 192.168.45.10 local 192.168.45.30 ikey 0.0.0.32 okey 0.0.0.42 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65
> 535
>
> [root at arch-linux ~]# cat /etc/swanctl/swanctl.conf | grep mark
> mark_in = 32
> mark_out = 42
>
>
> Let's run a ping from remote(10.10.10.1 pfSense) to local(10.10.10.2 Arch Linux) and capture the packets in the VTI interface and filter OUTPUT chain.
>
> No. Time Source Destination
> 87 43.788630 10.10.10.1 > 10.10.10.2 ICMP 84 Echo (ping) request id=0xb421, seq=43/11008, ttl=64 (reply in 88)
> 88 43.788641 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0xb421, seq=43/11008, ttl=64 (request in 87)
>
> The request arrives to the ip_vti1 interface and a reply is generated.
>
> No. Time Source Destination
> 144 60.693330 10.10.10.2 > 10.10.10.1 ICMP 116 Echo (ping) reply id=0xb421, seq=43/11008, ttl=64
>
> The reply is sent to the OUTPUT chain without encapsulation or encryption. It is like if the replies are not being seem by the IPSec stack.
>
> I am changing the interfaces settings as follows for another test 👇
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ens18
> net.ipv4.conf.ens18.disable_policy = 1
> net.ipv4.conf.ens18.rp_filter = 2
> net.ipv4.conf.ens18.disable_xfrm = 0
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ip_vti1
> net.ipv4.conf.ip_vti1.disable_policy = 1
> net.ipv4.conf.ip_vti1.rp_filter = 0
> net.ipv4.conf.ip_vti1.disable_xfrm = 0
>
> The results of the captures are the same as before.
>
> This time I will add iptables rules to manually MARK the packets and disable XFRM for both interfaces and |rp_filter| for the WAN (ens18).
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ens18
> net.ipv4.conf.ens18.disable_policy = 1
> net.ipv4.conf.ens18.rp_filter = 0
> net.ipv4.conf.ens18.disable_xfrm = 0
>
> [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ip_vti1
> net.ipv4.conf.ip_vti1.disable_policy = 1
> net.ipv4.conf.ip_vti1.rp_filter = 0
> net.ipv4.conf.ip_vti1.disable_xfrm = 0
>
> [root at arch-linux ~]# smangle
> -A PREROUTING -s 10.10.10.1/32 -d 10.10.10.2/32 -c 0 0 -j MARK --set-xmark 0x20/0xffffffff
> -A PREROUTING -s 10.10.10.2/32 -d 10.10.10.1/32 -c 0 0 -j MARK --set-xmark 0x2a/0xffffffff
>
> [root at arch-linux ~]# lmangle
> Chain PREROUTING (policy ACCEPT 78 packets, 5124 bytes)
> num pkts bytes target prot opt in out source destination
> 1 0 0 MARK all -- any any 10.10.10.1 10.10.10.2 MARK set 0x20
> 2 0 0 MARK all -- any any 10.10.10.2 10.10.10.1 MARK set 0x2a
> 3 181K 18M NFLOG all -- any any anywhere 10.10.10.0/30 nflog-group 5
>
> The results of the captures are the same as before. ☹
>
> You may ask if I have tested only changing |disable_xfrm| for |ens18| as suggested in your previous email and the answer is yes and the results were exactly the same as all the tests documented above and below.
>
> I can see the packets destined to the 10.10.10.2 ( Arch Linux ) being marked but the replies are not.
>
> [root at arch-linux ~]# lmangle
> Chain PREROUTING (policy ACCEPT 572 packets, 44220 bytes)
> num pkts bytes target prot opt in out source destination
> 1 66 5544 MARK all -- any any 10.10.10.1 10.10.10.2 MARK set 0x20
> 2 0 0 MARK all -- any any 10.10.10.2 10.10.10.1 MARK set 0x2a
> 3 181K 18M NFLOG all -- any any anywhere 10.10.10.0/30 nflog-group 5
>
>
> By the way, a couple of emails ago I have mentioned that when the packet is captured by NFLOG you can see the MARK. Here's one example where you can see the mark in the field NFULA_MARK.
>
> Frame 1: 144 bytes on wire (1152 bits), 144 bytes captured (1152 bits)
> Linux Netfilter NFLOG
> Family: IPv4 (2)
> Version: 0
> Resource id: 5
> TLV Type: NFULA_PACKET_HDR (1), Length: 8
> TLV Type: NFULA_PREFIX (10), Length: 5
> TLV Type: NFULA_IFINDEX_INDEV (4), Length: 8
> TLV Type: NFULA_MARK (2), Length: 8
> Length: 8
> .000 0000 0000 0010 = Type: NFULA_MARK (2)
> Value: 00000020
> TLV Type: NFULA_HWTYPE (15), Length: 6
> TLV Type: NFULA_HWLEN (17), Length: 6
> TLV Type: NFULA_HWHEADER (16), Length: 4
> TLV Type: NFULA_PAYLOAD (9), Length: 88
> Internet Protocol Version 4, Src: 10.10.10.1, Dst: 10.10.10.2
>
> I appreciate the help and clarification provided up to this point. Let me know if you have any more suggestions and I can happily test it.
>
> Tiago Stoco.
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Noel Kuntze
> *Sent:* Thursday, September 2, 2021 6:08 PM
> *To:* Tiago Stoco; Noel Kuntze; Tobias Brunner; users at lists.strongswan.org
> *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
>
> Hello Tiago,
>
> Thank you.
>
> > net.ipv4.conf.ens18.disable_xfrm = 1
>
> There's your problem.
> Set it to 0 (default value).
>
> Kind regards
> Noel
>
> Am 02.09.21 um 20:41 schrieb Tiago Stoco:
> > Hi Noel,
> >
> > So you're running openwrt on an Arch Linux kernel?
> >
> > No, I am running a pure Arch Linux virtual machine. Although, I have another VM with OpenWRT in my lab. Originally OpenWRT was being used but because NFLOG was not working and I needed packets flowing through the iptables captured to further troubleshoot an Arch box was spun up.
> >
> > Please pastebin the output of `sysctl net | grep disable`.
> >
> >
> > [root at arch-linux ~]# ./07.ipsec-check-int-sysctl.sh ip_vti1
> > net.ipv4.conf.ip_vti1.disable_policy = 1
> > net.ipv4.conf.ip_vti1.rp_filter = 0
> > net.ipv4.ip_forward = 1
> >
> > and as you requested 👇
> >
> > [root at arch-linux ~]# sysctl net | grep disable
> > net.core.high_order_alloc_disable = 0
> > net.ipv4.conf.all.disable_policy = 0
> > net.ipv4.conf.all.disable_xfrm = 0
> > net.ipv4.conf.default.disable_policy = 0
> > net.ipv4.conf.default.disable_xfrm = 0
> > net.ipv4.conf.ens18.disable_policy = 1
> > net.ipv4.conf.ens18.disable_xfrm = 1
> > net.ipv4.conf.ip_vti0.disable_policy = 0
> > net.ipv4.conf.ip_vti0.disable_xfrm = 0
> > net.ipv4.conf.ip_vti1.disable_policy = 1
> > net.ipv4.conf.ip_vti1.disable_xfrm = 0
> > net.ipv4.conf.lo.disable_policy = 1
> > net.ipv4.conf.lo.disable_xfrm = 1
> > net.ipv6.conf.all.disable_ipv6 = 0
> > net.ipv6.conf.all.disable_policy = 0
> > net.ipv6.conf.default.disable_ipv6 = 0
> > net.ipv6.conf.default.disable_policy = 0
> > net.ipv6.conf.ens18.disable_ipv6 = 0
> > net.ipv6.conf.ens18.disable_policy = 0
> > net.ipv6.conf.ip_vti1.disable_ipv6 = 0
> > net.ipv6.conf.ip_vti1.disable_policy = 0
> > net.ipv6.conf.lo.disable_ipv6 = 0
> > net.ipv6.conf.lo.disable_policy = 0
> >
> > Many Thanks.
> >
> > Tiago Stoco.
> >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > *From:* Noel Kuntze
> > *Sent:* Thursday, September 2, 2021 3:28 AM
> > *To:* Tiago Stoco; Noel Kuntze; Tobias Brunner; users at lists.strongswan.org
> > *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
> >
> > Hello Tiago,
> >
> > > Linux 5.13.12-arch1-1
> >
> > H
> >
> > > According to my understanding, the reply should be marked, dealt with the IPSec stack, and tunneled to the peer since it is on the VTI interface. Please correct me if I am wrong.
> >
> > Please pastebin the output of `sysctl net | grep disable`.
> >
> > Kind regards
> > Noel
> >
> > Am 02.09.21 um 08:18 schrieb Tiago Stoco:
> > > Hi Noel,
> > >
> > > I have disabled the forecast plugin and MARK rules aren't being added to iptables anymore.
> > >
> > > The following plugins are disabled at the moment :
> > >
> > > [root at arch-linux ~]# cat /etc/strongswan.conf
> > > ...
> > > bypass-lan {
> > > load = no
> > > }
> > > connmark {
> > > load = no
> > > }
> > > forecast {
> > > load = no
> > > }
> > > ...
> > >
> > > When a capture is run on the VTI device I can see the pings from the pfSense arriving and the reply 👇
> > >
> > > No. Time Source Destination
> > > 5 2.050455 10.10.10.1 > 10.10.10.2 ICMP 84 Echo (ping) request id=0xc4f5, seq=2/512, ttl=64 (reply in 6)
> > > 6 2.050467 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0xc4f5, seq=2/512, ttl=64 (request in 5)
> > >
> > > Strange is the fact that I see the replies in the filter OUTPUT chain not encapsulated nor encrypted 👇
> > >
> > > No. Time Source Destination
> > > 11 9.653361 10.10.10.2 > 10.10.10.1 ICMP 116 Echo (ping) reply id=0x80f2, seq=2/512, ttl=64
> > >
> > > According to my understanding, the reply should be marked, dealt with the IPSec stack, and tunneled to the peer since it is on the VTI interface. Please correct me if I am wrong.
> > >
> > > In regards to using an XFRM, the problem is that OpenWRT does not support XFRM devices. A newer release will support but the current is not available yet.
> > >
> > > [root at arch-linux ~]# ipsec statusall
> > > Status of IKE charon daemon (strongSwan 5.9.3, Linux 5.13.12-arch1-1, x86_64):
> > > uptime: 9 hours, since Sep 01 21:37:46 2021
> > > malloc: sbrk 3137536, mmap 0, used 1156720, free 1980816
> > > worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
> > > loaded plugins: charon ldap pkcs11 aes des rc2 sha2 sha3 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey p
> > > kcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp curve25519 agent chapoly xcbc cmac hmac ntru drbg newho
> > > pe bliss curl sqlite attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-a
> > > ka-3gpp2 eap-simaka-pseudonym eap-simaka-reauth eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-p
> > > eap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp radattr unity counters
> > > Listening IP addresses:
> > > 192.168.45.30
> > > 10.10.10.2
> > > Connections:
> > > ipseclab: 192.168.45.30...192.168.45.10 IKEv2, dpddelay=10s
> > > ipseclab: local: [ipsec-lab-openwrt] uses pre-shared key authentication
> > > ipseclab: remote: [ipsec-lab-pfsense] uses pre-shared key authentication
> > > con1: child: 10.10.10.0/30 === 10.10.10.0/30 TUNNEL, dpdaction=restart
> > > Security Associations (1 up, 0 connecting):
> > > ipseclab[4]: ESTABLISHED 2 hours ago, 192.168.45.30[ipsec-lab-openwrt]...192.168.45.10[ipsec-lab-pfsense]
> > > ipseclab[4]: IKEv2 SPIs: bbbb8b7b5c0ccf5b_i 9673036f29862401_r*, rekeying in 4 hours
> > > ipseclab[4]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
> > > con1{12}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cef2dc29_i ccb51306_o
> > > con1{12}: AES_GCM_16_256/MODP_2048, 0 bytes_i, 0 bytes_o, rekeying in 14 minutes
> > > con1{12}: 10.10.10.0/30 === 10.10.10.0/30
> > >
> > > Many Thanks !!
> > >
> > >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > *From:* Noel Kuntze
> > > *Sent:* Wednesday, September 1, 2021 1:23 AM
> > > *To:* Tiago Stoco; Noel Kuntze; Tobias Brunner; users at lists.strongswan.org
> > > *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
> > >
> > > Hi Tiago,
> > >
> > > Try disabling the forecast plugin too, please.
> > >
> > > With VTIs, you shouldn't need to manually mark the packets.
> > >
> > > Btw, if you use an XFRM interface instead, you won't have as many problems because the field used for
> > > typing the interface and the policies together isn't used for anything else (In comparison to the fwmark field,
> > > which as you can see can be used for all sorts of things).
> > >
> > > >
> > > > I will add to the thread later but NFLOG is able to capture MARKS in the packets. The packets captured by NFLOG has a special section with quite useful information.
> > > > ...
> > > > Linux Netfilter NFLOG
> > > > Family: IPv4 (2)
> > > > Version: 0
> > > > Resource id: 7
> > > > TLV Type: NFULA_PACKET_HDR (1), Length: 8
> > > > ...
> > >
> > > That doesn't look like a mark field to me. It'd need to be 32 bit in length and have the same value as the (fw)mark field in iptables.
> > >
> > > Kind regards
> > > Noel
> > >
> > >
> > > Am 01.09.21 um 01:46 schrieb Tiago Stoco:
> > > > Hi Noel,
> > > >
> > > > Thanks for the help.
> > > >
> > > > I have moved the route for IPSec back into the main routing table.
> > > >
> > > > [root at arch-linux ~]# ip route
> > > > default via 192.168.45.1 dev ens18
> > > > 10.10.10.0/30 dev ip_vti1 scope link src 10.10.10.2
> > > > 192.168.45.0/24 dev ens18 proto kernel scope link src 192.168.45.30
> > > >
> > > > [root at arch-linux ~]# ip route show table 220
> > > > [root at arch-linux ~]#
> > > >
> > > > And the connmark plugin has been disabled.
> > > >
> > > > [root at arch-linux ~]# ipsec statusall | grep -i connmark
> > > > [root at arch-linux ~]#
> > > >
> > > > Also, I have noticed that even after the connmark plugin is disabled some mark rules are added to iptables.
> > > >
> > > > [root at arch-linux ~]# ipsec stop
> > > > Stopping strongSwan IPsec...
> > > > [root at arch-linux ~]# iptables-save | grep -i mark
> > > > [root at arch-linux ~]#
> > > >
> > > > [root at arch-linux ~]# ipsec start
> > > > Starting strongSwan 5.9.3 IPsec [starter]...
> > > > [root at arch-linux ~]# swanctl --load-all
> > > > plugin 'mysql' failed to load: libmariadb.so.3: cannot open shared object file: No such file or directory
> > > > loaded ike secret 'ike-0'
> > > > no authorities found, 0 unloaded
> > > > no pools found, 0 unloaded
> > > > loaded connection 'ipseclab'
> > > > successfully loaded 1 connections, 0 unloaded
> > > > [root at arch-linux ~]# iptables-save | grep -i mark
> > > > -A PREROUTING -d 10.10.10.0/30 -j MARK --set-xmark 0x2a/0xffffffff
> > > > -A PREROUTING -s 192.168.45.10/32 -d 192.168.45.30/32 -p esp -m esp --espspi 3419086685 -j MARK --set-xmark 0x2a/0xfffffff
> > > > f
> > > > -A OUTPUT -d 10.10.10.0/30 -j MARK --set-xmark 0x2a/0xffffffff
> > > >
> > > >
> > > > The scenario is the same at the moment :
> > > >
> > > > pings from 10.10.10.1 ( pfSense ) to 10.10.10.2 ( Linux ) ✅ seem as received by Linux
> > > > pings from 10.10.10.2 ( Linux ) to 10.10.10.1 ( pfSense ) ❌ seem as Errors NoRoute
> > > > ip_vti1: ip/ip remote 192.168.45.10 local 192.168.45.30 ttl inherit nopmtudisc key 42
> > > > RX: Packets Bytes Errors CsumErrs OutOfSeq Mcasts
> > > > 66095 5551980 0 0 0 0
> > > > TX: Packets Bytes Errors DeadLoop NoRoute NoBufs
> > > > 0 0 133788 0 133788 0
> > > >
> > > > I am at work and not able to capture packets to further investigate but will do as soon possible.
> > > >
> > > > [root at arch-linux ~]# ipsec statusall
> > > > Status of IKE charon daemon (strongSwan 5.9.3, Linux 5.13.12-arch1-1, x86_64):
> > > > uptime: 9 minutes, since Sep 01 00:24:27 2021
> > > > malloc: sbrk 2936832, mmap 0, used 1328288, free 1608544
> > > > worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 115
> > > > loaded plugins: charon ldap pkcs11 aes des rc2 sha2 sha3 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey p
> > > > kcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp curve25519 agent chapoly xcbc cmac hmac ntru drbg newho
> > > > pe bliss curl sqlite attr kernel-netlink resolve socket-default forecast farp stroke vici updown eap-identity eap-sim eap-
> > > > aka eap-aka-3gpp2 eap-simaka-pseudonym eap-simaka-reauth eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-t
> > > > tls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp radattr unity counters
> > > > Listening IP addresses:
> > > > 192.168.45.30
> > > > 10.10.10.2
> > > > Connections:
> > > > ipseclab: 192.168.45.30...192.168.45.10 IKEv2, dpddelay=10s
> > > > ipseclab: local: [ipsec-lab-openwrt] uses pre-shared key authentication
> > > > ipseclab: remote: [ipsec-lab-pfsense] uses pre-shared key authentication
> > > > con1: child: 10.10.10.0/30 === 10.10.10.0/30 TUNNEL, dpdaction=restart
> > > > Security Associations (2 up, 0 connecting):
> > > > ipseclab[54]: ESTABLISHED 9 seconds ago, 192.168.45.30[ipsec-lab-openwrt]...192.168.45.10[ipsec-lab-pfsense]
> > > > ipseclab[54]: IKEv2 SPIs: 840c3fd77eb0efee_i b3f6cec233130e6a_r*, rekeying in 6 hours
> > > > ipseclab[54]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
> > > > con1{54}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: c206ad8c_i c631ebba_o
> > > > con1{54}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying in 48 minutes
> > > > con1{54}: 10.10.10.0/30 === 10.10.10.0/30
> > > > ipseclab[53]: ESTABLISHED 19 seconds ago, 192.168.45.30[ipsec-lab-openwrt]...192.168.45.10[ipsec-lab-pfsense]
> > > > ipseclab[53]: IKEv2 SPIs: 133d2066ebf6a358_i d8da41bca97601ca_r*, rekeying in 6 hours
> > > > ipseclab[53]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
> > > > con1{53}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cb169570_i c35823e7_o
> > > > con1{53}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying in 51 minutes
> > > > con1{53}: 10.10.10.0/30 === 10.10.10.0/30
> > > >
> > > > I will add to the thread later but NFLOG is able to capture MARKS in the packets. The packets captured by NFLOG has a special section with quite useful information.
> > > > ...
> > > > Linux Netfilter NFLOG
> > > > Family: IPv4 (2)
> > > > Version: 0
> > > > Resource id: 7
> > > > TLV Type: NFULA_PACKET_HDR (1), Length: 8
> > > > ...
> > > >
> > > > Once again, thanks for the help.
> > > >
> > > > Tiago Stoco.
> > > >
> > > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > > *From:* Noel Kuntze
> > > > *Sent:* Tuesday, August 31, 2021 2:20 PM
> > > > *To:* Tiago Stoco; Tobias Brunner; users at lists.strongswan.org
> > > > *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
> > > >
> > > > Hello Tiago,
> > > >
> > > > > And, I have moved the route for the VTI to table 220 because it seems to be the right way to config routed based IPSec VPN.
> > > > >
> > > > > [root at arch-linux ~]# ip rule
> > > > > 0: from all lookup local
> > > > > 220: from all lookup 220
> > > > > 32766: from all lookup main
> > > > > 32767: from all lookup default
> > > >
> > > > Don't do that.
> > > > 220 is managed by strongSwan. Keep them in the main table.
> > > >
> > > > > -A PREROUTING -d 10.10.10.0/30 -c 2352 230776 -j MARK --set-xmark 0x2a/0xffffffff
> > > > > -A OUTPUT -d 10.10.10.0/30 -c 3605 336028 -j MARK --set-xmark 0x2a/0xffffffff
> > > >
> > > > Disable the connmark plugin.
> > > >
> > > >
> > > > > I have added a few more NFLOG captures into my iptables and I am a bit confused with the results.
> > > > >
> > > > > A tcpdump capture in the VTI interface with a ping from the remote ( pfSense - 10.10.10.1 ) shows :
> > > > >
> > > > > No Time Source Destination
> > > > >
> > > > > 1 0.000000 10.10.10.1 10.10.10.2 ICMP 84 Echo (ping) request id=0x9877, seq=471/55041, ttl=64 (reply in 2)
> > > > > 2 0.000038 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0x9877, seq=471/55041, ttl=64 (request in 1)
> > > > >
> > > > > I do not see the IPSec MARK in these packets.
> > > >
> > > > They are only visible in the output of the TRACE target. I suspect they are note copied into the buffer passed to the applications.
> > > >
> > > > >
> > > > > Also, the NAT chain is not having packets passing through it.
> > > > >
> > > > > [root at arch-linux ~]# snat
> > > > > -P PREROUTING ACCEPT -c 0 0
> > > > > -P INPUT ACCEPT -c 0 0
> > > > > -P OUTPUT ACCEPT -c 0 0
> > > > > -P POSTROUTING ACCEPT -c 0 0
> > > > > -A PREROUTING -c 0 0 -j NFLOG --nflog-group 9
> > > > >
> > > > > That is odd cause I am not able to manipulate the packets.
> > > > >
> > > > > I will run a ping from the local Linux (10.10.10.2) and see how the packets are flowing through the iptables chains and will update in another email.
> > > > >
> > > >
> > > > The *nat table is only consulted for the first packet of a connection.
> > > >
> > > > Kind regards
> > > > Noel
> > > >
> > > >
> > > > Am 31.08.21 um 17:22 schrieb Tiago Stoco:
> > > > > Hi Tobias,
> > > > >
> > > > > First of all, THANKS for replying and clarifying some settings.
> > > > >
> > > > > I have completely disabled the bypass-lan plugin since I do not have a use for it right now.
> > > > >
> > > > > [root at arch-linux ~]# cat /etc/strongswan.conf
> > > > > ...
> > > > > plugins {
> > > > > include strongswan.d/charon/*.conf
> > > > > bypass-lan {
> > > > > load = no
> > > > > }
> > > > > ...
> > > > >
> > > > > And, I have moved the route for the VTI to table 220 because it seems to be the right way to config routed based IPSec VPN.
> > > > >
> > > > > [root at arch-linux ~]# ip rule
> > > > > 0: from all lookup local
> > > > > 220: from all lookup 220
> > > > > 32766: from all lookup main
> > > > > 32767: from all lookup default
> > > > >
> > > > > [root at arch-linux ~]# ip r s t 220
> > > > > 10.10.10.0/30 via 10.10.10.2 dev ip_vti1 src 10.10.10.2
> > > > >
> > > > > [root at arch-linux ~]# ip route
> > > > > default via 192.168.45.1 dev ens18
> > > > > 192.168.45.0/24 dev ens18 proto kernel scope link src 192.168.45.30
> > > > >
> > > > > I am going to add some more details of my configs because the TX Errors NoRoute are still present.
> > > > >
> > > > > 7: ip_vti1 at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN group default qlen 1000
> > > > > link/ipip 192.168.45.30peer 192.168.45.10promiscuity 0 minmtu 0 maxmtu 0
> > > > > vti remote 192.168.45.10 local 192.168.45.30 ikey 0.0.0.42 okey 0.0.0.42 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> > > > > inet 10.10.10.2peer 10.10.10.1/32 scope global ip_vti1
> > > > > valid_lft forever preferred_lft forever
> > > > > inet6 fe80::5efe:c0a8:2d1e/64 scope link
> > > > > valid_lft forever preferred_lft forever
> > > > >
> > > > > I can also see that the IPSec added some rules to MARK packets in my iptables.
> > > > >
> > > > > -A PREROUTING -d 10.10.10.0/30 -c 2352 230776 -j MARK --set-xmark 0x2a/0xffffffff
> > > > > -A OUTPUT -d 10.10.10.0/30 -c 3605 336028 -j MARK --set-xmark 0x2a/0xffffffff
> > > > >
> > > > > The counters confirms that the packets are being marked. I am not sure if I should keep the MARK in iptables or remove it allowing routing decisions to send the packets to the VTI device that will MARK the packets but according to my understanding it should not matter.
> > > > >
> > > > > [root at arch-linux ~]# ip xfrm policy
> > > > > src 0.0.0.0/0 dst 0.0.0.0/0
> > > > > socket in priority 0 ptype main
> > > > > src 0.0.0.0/0 dst 0.0.0.0/0
> > > > > socket out priority 0 ptype main
> > > > > src 0.0.0.0/0 dst 0.0.0.0/0
> > > > > socket in priority 0 ptype main
> > > > > src 0.0.0.0/0 dst 0.0.0.0/0
> > > > > socket out priority 0 ptype main
> > > > > src ::/0 dst ::/0
> > > > > socket in priority 0 ptype main
> > > > > src ::/0 dst ::/0
> > > > > socket out priority 0 ptype main
> > > > > src ::/0 dst ::/0
> > > > > socket in priority 0 ptype main
> > > > > src ::/0 dst ::/0
> > > > > socket out priority 0 ptype main
> > > > >
> > > > > Above are the policies installed. Again, because it is a routed base VPN seems correct.
> > > > >
> > > > > [root at arch-linux ~]# ip xfrm state
> > > > > src 192.168.45.30 dst 192.168.45.10
> > > > > proto esp spi 0xc2239b57 reqid 1 mode tunnel
> > > > > replay-window 0 flag af-unspec
> > > > > mark 0x2a/0xffffffff
> > > > > aead rfc4106(gcm(aes)) 0x264acee3119a4e523af2fbf5905b50c5acc1f7be9079ff23ffa2c6473a9c507fe1ae936b 128
> > > > > anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
> > > > > src 192.168.45.10 dst 192.168.45.30
> > > > > proto esp spi 0xc661b9e5 reqid 1 mode tunnel
> > > > > replay-window 32 flag af-unspec
> > > > > aead rfc4106(gcm(aes)) 0x69a86fa6ca9448bece6ffdff77893f0e9ce5ebef604040f681b5cdd2d5976438ed005df1 128
> > > > > anti-replay context: seq 0x656, oseq 0x0, bitmap 0xffffffff
> > > > >
> > > > > I have added a few more NFLOG captures into my iptables and I am a bit confused with the results.
> > > > >
> > > > > A tcpdump capture in the VTI interface with a ping from the remote ( pfSense - 10.10.10.1 ) shows :
> > > > >
> > > > > No Time Source Destination
> > > > >
> > > > > 1 0.000000 10.10.10.1 10.10.10.2 ICMP 84 Echo (ping) request id=0x9877, seq=471/55041, ttl=64 (reply in 2)
> > > > > 2 0.000038 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0x9877, seq=471/55041, ttl=64 (request in 1)
> > > > >
> > > > > I do not see the IPSec MARK in these packets.
> > > > > The reply packets end up in the OUTPUT chain marked but not encrypted as an ESP packet. By the way I do not see the replies even being encapsulated at all by IPSec.
> > > > >
> > > > > Also, the NAT chain is not having packets passing through it.
> > > > >
> > > > > [root at arch-linux ~]# snat
> > > > > -P PREROUTING ACCEPT -c 0 0
> > > > > -P INPUT ACCEPT -c 0 0
> > > > > -P OUTPUT ACCEPT -c 0 0
> > > > > -P POSTROUTING ACCEPT -c 0 0
> > > > > -A PREROUTING -c 0 0 -j NFLOG --nflog-group 9
> > > > >
> > > > > That is odd cause I am not able to manipulate the packets.
> > > > >
> > > > > I will run a ping from the local Linux (10.10.10.2) and see how the packets are flowing through the iptables chains and will update in another email.
> > > > >
> > > > > In the meantime, if someone sees something that I am missing. Please let me know.
> > > > >
> > > > > Many Thanks.
> > > > >
> > >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > > > *From:* Tobias Brunner <tobias at strongswan.org>
> > > > > *Sent:* Tuesday, August 31, 2021 5:51 AM
> > > > > *To:* Tiago Stoco <tmsblink at msn.com>; users at lists.strongswan.org <users at lists.strongswan.org>
> > > > > *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors NoRoute
> > > > > Hi Tiago,
> > > > >
> > > > >> Pings from the Linux system are being seem as errors NoRoute by the tunnel. > ...
> > > > >> Shunted Connections:
> > > > >> Bypass LAN 10.10.10.0/30: 10.10.10.0/30 === 10.10.10.0/30 PASS
> > > > >
> > > > > The reason is most likely this passthrough IPsec policy installed by the
> > > > > bypass-lan plugin for the subnet that is reachable (according to the
> > > > > main routing table) via ip_vti1. For a ping from 10.10.10.2 to
> > > > > 10.10.10.1, the VTI interface won't find an IPsec policy to protect the
> > > > > packet (the passthrough policy has a higher priority), so it gets dropped.
> > > > >
> > > > > To avoid that, either install the routes via VTI in table 220 (which is
> > > > > ignored by the bypass-lan plugin automatically), exclude the VTI
> > > > > interface explicitly via charon.plugins.bypass-lan.interfaces_ignore, or
> > > > > just disable the bypass-lan plugin completely if you don't need it.
> > > > >
> > > > > Regards,
> > > > > Tobias
> > > >
> > >
> > >
> >
> >
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.strongswan.org/pipermail/users/attachments/20210903/3f784e02/attachment-0001.sig>
More information about the Users
mailing list