[strongSwan] GRE over IPSec issue: ping: sendmsg: Invalid argument

[Florin Andrei]

Two instances, both running CentOS 6 and strongSwan 5.4.0. The kernel is 
2.6.32-754.17.1.el6.x86_64

Both instances are behind NAT (AWS, separate VPCs, different EIPs 
assigned to each instance - the two private IP ranges are completely 
separate). This is ipsec.conf on one instance (the other instance's 
config is similar, just mirrored left/right):

#######################################
config setup
   strictcrlpolicy=yes
   uniqueids = yes

conn %default
   authby = psk
   auto = start
   compress = yes
   dpdaction = restart
   dpddelay = 5s
   dpdtimeout = 30s
   esp = aes256-sha512-modp4096
   forceencaps = yes
   fragmentation = yes
   ike = aes256-sha512-modp4096
   keyexchange = ikev2
   type = tunnel

conn site1-site2
   left = %any
   leftid = site1
   leftsubnet = 10.0.1.254/32
   right = public-IP-of-other-instance
   rightid = site2
   rightsubnet = 10.0.2.254/32
#######################################

leftsubnet and rightsubnet are the private IPs assigned to eth0 on the 
VPN instances.

Once the IPSec tunnel is established, I can ping eth0 of instance 
site2-vpn from instance site1-vpn, and viceversa. So the IPSec tunnel 
works fine. I've used strongSwan before, I'm familiar with this part.

#######################################
[root at site1-vpn ~]# ping -n 10.0.2.254
PING 10.0.2.254 (10.0.2.254) 56(84) bytes of data.
64 bytes from 10.0.2.254: icmp_seq=1 ttl=64 time=0.606 ms
64 bytes from 10.0.2.254: icmp_seq=2 ttl=64 time=0.690 ms
64 bytes from 10.0.2.254: icmp_seq=3 ttl=64 time=0.685 ms
#######################################

My initial goal is to create a GRE tunnel between the two instances, 
then run dynamic routing (BGP, OSPF, whatever works) and let dynamic 
routing take care of adding/removing routes between VPN nodes (which are 
default gateways to their local networks). The ultimate goal is to have 
a mesh of VPN nodes with multiple tunnels in between and dynamic 
routing. I want strongSwan to handle the minimum amount of routing 
needed to create the GRE tunnels, and then let dynamic routing handle 
the bulk of the routing tables on top of GRE.

But first, I need to establish one GRE tunnel between two nodes.

This is how I create the GRE tunnel (the local and remote IPs are the 
eth0 addresses on the VPN instances):

#######################################
# site1-vpn
ip tunnel add gre0 local 10.0.1.254 remote 10.0.2.254 mode gre ttl 255
ip link set gre0 up
ip addr add 10.2.0.1/24 dev gre0

# site2-vpn
ip tunnel add gre0 local 10.0.2.254 remote 10.0.1.254 mode gre ttl 255
ip link set gre0 up
ip addr add 10.2.0.2/24 dev gre0
#######################################

But when I try to ping the other end of the tunnel, it does not work:

#######################################
[root at site1-vpn ~]# ping -n 10.2.0.2
PING 10.2.0.2 (10.2.0.2) 56(84) bytes of data.
ping: sendmsg: Invalid argument
ping: sendmsg: Invalid argument
ping: sendmsg: Invalid argument
#######################################

tcpdump on the other instance shows no activity - no packets are even 
being sent to the other side via the IPSec tunnel. But the IPSec tunnel 
continues to work, I can ping eth0 IPs across the tunnel just fine.

This is networking on one instance (the other is similar, just 
mirrored):

#######################################
[root at site1-vpn ~]# ip route
10.0.1.128/25 dev eth0  proto kernel  scope link  src 10.0.1.254
10.2.0.0/24 dev gre0  proto kernel  scope link  src 10.2.0.1
default via 10.0.1.129 dev eth0

[root at site1-vpn ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast 
state UP qlen 1000
     link/ether 02:96:54:76:52:80 brd ff:ff:ff:ff:ff:ff
     inet 10.0.1.254/25 brd 10.0.1.255 scope global eth0
     inet6 fe80::96:54ff:fe76:5280/64 scope link
        valid_lft forever preferred_lft forever
3: gre0: <NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN
     link/gre 0.0.0.0 brd 0.0.0.0
     inet 10.2.0.1/24 scope global gre0
4: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 
1000
     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
#######################################

One more thing - when I run "ip tunnel add" I get this warning (I don't 
know whether it matters or not):

#######################################
[root at site1-vpn ~]# ip tunnel add gre0 local 10.0.1.254 remote 
10.0.2.254 mode gre ttl 255
ioctl: File exists
#######################################

-- 
Florin Andrei
http://florin.myip.org/