[strongSwan] OS X IPSec/L2TP and strongSwan results in INVALID_HASH_INFORMATION

[Benoit Foucher]

Hi,

More investigations on this one and I found the reason of this INVALID_HASH_INFORMATION error. As you might have guessed strongSwan and raccoon aren't computing the hash on the same data, strongSwan includes the NAT-OA in the hash computation, raccoon doesn't. If I change strongSwan's code to not include it in the hash computation I can establish the IPsec connection. 

Specifically, in pluto/ipsec_doi.c (around line 5145), I move the following code:

		/* Compute reply HASH(2) and insert in output */
		(void)quick_mode_hash12(r_hashval, r_hash_start, md->rbody.cur
			, st, &st->st_msgid, TRUE);

before this block:

		if ((st->nat_traversal & NAT_T_WITH_NATOA)
		&& (st->nat_traversal & LELEM(NAT_TRAVERSAL_NAT_BHND_ME))
		&& (st->st_esp.attrs.encapsulation == ENCAPSULATION_MODE_TRANSPORT))
		{
			/** Send NAT-OA if our address is NATed and if we use Transport Mode */
			if (!nat_traversal_add_natoa(ISAKMP_NEXT_NONE, &md->rbody, md->st))
			{
				return STF_INTERNAL_ERROR;
			}
		}

Could you confirm whether or not this is a raccoon or strongSwan issue?

Now, the IPSec SA is established but there's still one issue with the routing :(. My strongSwan server receives the L2TP packets on port 1701 but the replies aren't getting back through the tunnel (I can see the l2tp udp packets with tcpdump). I suspect this is because the xfrm policies are bogus. Here's the configuration I'm using 

conn rwl2tp
        esp=aes128-sha1
        ike=aes128-sha-modp1024
        keyexchange=ikev1
        keyingtries=3
        type=transport
        left=10.1.1.1
        leftsubnet=17.12.21.19/32
        leftprotoport=17/1701
        right=%any
        rightprotoport=17/%any
        rightsubnetwithin=0.0.0.0/0
        authby=psk
        pfs=no
        auto=add

17.12.21.19 is the public IP address of the strongSwan server. The NAT address is 10.1.1.1. 

The xfrm policy that get installed are with the public IP address:

  src 17.12.21.19/32 dst 93.18.20.78/32 proto udp sport 1701 dport 65139 
          dir out priority 1792 ptype main 
          tmpl src 0.0.0.0 dst 0.0.0.0
                  proto esp reqid 16452 mode transport
  src 93.18.20.78/32 dst 17.12.21.19/32 proto udp sport 65139 dport 1701 
          dir in priority 1792 ptype main 
          tmpl src 0.0.0.0 dst 0.0.0.0
                  proto esp reqid 16452 mode transport

I guess that's why the L2TP traffic isn't going back through the tunnel, shouldn't the policies instead use the 10.1.1.1 address? Is this a problem with raccoon requiring letsubnet to be set to 17.12.21.19/32 and not checking for the NAT-OA address sent by pluto?

Thanks for your help!

Cheers,
Benoit.

On Dec 3, 2010, at 3:59 PM, Benoit Foucher wrote:

> Hi,
> 
> Ok, next issue :). I'm trying to setup an OS X client IPSec/L2TP connection to strongSwan 4.5.0.
> 
> The strongSwan server and the OS X client are both behind a NAT. I managed to find the configuration to get the tunnel establishment to pass phase 1 but it fails in phase 2. The OS X client (raccoon) fails to match its computed HASH(2) with strongSwan's hash passed with the STATE_QUICK_R0 message. I've attached the strongSwan debug traces and raccoon debug traces to this email. Any ideas why raccoon and strongSwan don't agree on the hash value?
> 
> Someone reported a similar issue last month and indicated that things were working when the strongSwan server was NOT behind a NAT but failed when it was behind a NAT.
> 
> Here's the config I'm using:
> 
> conn rw
>        esp=aes128-sha1
>        ike=aes128-sha-modp1024
>        keyexchange=ikev1
>        keyingtries=3
>        type=transport
>        left=%defaultroute
>        leftsubnet=aa.aa.aa.aa/32
>        leftprotoport=17/1701
>        right=%any
>        rightprotoport=17/%any
>        rightsubnetwithin=0.0.0.0/0
>        authby=psk
>        pfs=no
>        compress=no
>        auto=add
> 
> Cheers,
> Benoit.
> 
> <racoon.log><pluto2.log>