<div dir="ltr">Thanks Thomas. I (sort of) understand the issue. A few followup questions:<div><br></div><div>1) If the packets are delivered out-of-order. Is there any specific reason that they aren't just dropped so that the higher level tcp level would just re-send? <div><br></div><div>2) With regards to talking a look at the packet captures; aside from knowing that they arrived out of order, anything in particular I should be looking for? I realize this might be too general a question so no worries if an answer isn't possible.</div><div><br></div><div><b><u>On a positive note</u></b>, right after noticing that the issue was related to replay window (and before i received your explanation) I did re-start strongswan with charon.replay_window = 0 (apparently that disables it) and so far the traffic has not paused over the tunnel. I'll also try increasing the replay_window at the next opportunity.</div><div><br></div><div>thanks again to both you & noel for answering on a weekend!<br><br></div><div>mahesh</div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 13, 2015 at 5:18 PM, Thomas Egerer <span dir="ltr"><<a href="mailto:hakke_007@gmx.de" target="_blank">hakke_007@gmx.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 12/13/2015 10:18 PM, Mahesh Neelakanta wrote:<br>
</span><span class="">> Thomas, the vpn paused and I ran the IP spi command in looks like the<br>
> replay-window keeps increasing. Any ideas what that means?<br>
</span>Yes, surely. The ip-xfrm framework uses a sliding window for replay<br>
detection. This means only a certain number of packets (in your case 32)<br>
less then the largest sequence number received are accepted. All packets<br>
below that limit are dropped (increasing the replay-window counter).<br>
This means your ESP-packets were reorderd and arrive in a different<br>
order than they were sent. Depending on your underlying (encrypted)<br>
traffic this can heal.<br>
To take countermeasures, you may want to increase your replay window:<br>
a) use the global charon.replay_window from strongswan.conf [1]<br>
b) use the connection specific ipsec.conf option replay_window<br>
(available since strongswan 5.2.0) [2].<br>
If this does not help, you can perform further investigation: take the<br>
broken tunnel check if any inbound packets are received, or if all of<br>
them are dropped. This can be done by running the iproute command 'ip -s<br>
x s s spi <tunnel_spi>' prior to a 'tcpdump -w <output-file> -i<br>
<interface>' for an arbitrary time (let's say one minute) and another<br>
iproute2-command as described above *immediately* after tcpdump was<br>
stopped. You can then analyze your received ESP-packets againts the<br>
number of replay-window errors before and after the capturing. Also: all<br>
inbound packets not dropped show up -- as Noel already pointed out -- in<br>
plain text again, so you can compare your results from iproute to your<br>
pcap file.<br>
<br>
Cheers,<br>
Thomas<br>
<br>
[1] <a href="https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf" rel="noreferrer" target="_blank">https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf</a><br>
[2] <a href="https://wiki.strongswan.org/projects/strongswan/wiki/ConnSection" rel="noreferrer" target="_blank">https://wiki.strongswan.org/projects/strongswan/wiki/ConnSection</a><br>
<span class="">><br>
> proto esp spi 0xc6ff382c(3338614828) reqid 2(0x00000002) mode tunnel<br>
> replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)<br>
> auth-trunc hmac(sha1) 0xc609a31c3e5b7d6fa5267737c759fed017d2d6ea<br>
> (160 bits) 96<br>
> enc cbc(aes) 0x4fba8977e230c1155780f03a19b90111 (128 bits)<br>
> lifetime config:<br>
> limit: soft (INF)(bytes), hard (INF)(bytes)<br>
> limit: soft (INF)(packets), hard (INF)(packets)<br>
> expire add: soft 3600(sec), hard 3600(sec)<br>
> expire use: soft 0(sec), hard 0(sec)<br>
> lifetime current:<br>
> 676746342(bytes), 703241(packets)<br>
> add 2015-12-13 20:39:21 use 2015-12-13 20:39:21<br>
> stats:<br>
</span>> * replay-window 533 replay 0 failed 0*<br>
<span class="">><br>
> On Sun, Dec 13, 2015 at 3:50 PM, Mahesh Neelakanta <<a href="mailto:neelakanta@gmail.com">neelakanta@gmail.com</a>><br>
> wrote:<br>
><br>
>> Thanks Thomas. I was able to run the "ip" command but it does look like<br>
>> (as you mentioned) that CONFIG_XFRM_STATISTICS is disabled (this is the<br>
>> amazon ubuntu 12.04 AMI). I'll try a newer release of amazon's own linux to<br>
>> see if it has it installed before trying a kernel recompile. Right now the<br>
>> ip command shows no errors (but i've restarted vpn) so i'll await it to<br>
>> hang again.<br>
>><br>
>> ip -s x s s spi 0xc6ff382c<br>
>><br>
>> proto esp spi 0xc6ff382c(3338614828) reqid 2(0x00000002) mode<br>
>> tunnel<br>
>> replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)<br>
>> auth-trunc hmac(sha1) 0xc609a31c3e5b7d6fa5267737c759fed017d2d6ea<br>
>> (160 bits) 96<br>
>> enc cbc(aes) 0x4fba8977e230c1155780f03a19b90111 (128 bits)<br>
>> lifetime config:<br>
>> limit: soft (INF)(bytes), hard (INF)(bytes)<br>
>> limit: soft (INF)(packets), hard (INF)(packets)<br>
>> expire add: soft 3600(sec), hard 3600(sec)<br>
>> expire use: soft 0(sec), hard 0(sec)<br>
>> lifetime current:<br>
>> 183887577(bytes), 191174(packets)<br>
>> add 2015-12-13 20:39:21 use 2015-12-13 20:39:21<br>
</span>>> * stats:*<br>
>> * replay-window 0 replay 0 failed 0*<br>
<div class="HOEnZb"><div class="h5">>><br>
>><br>
>> On Sun, Dec 13, 2015 at 3:23 PM, Thomas Egerer <<a href="mailto:hakke_007@gmx.de">hakke_007@gmx.de</a>> wrote:<br>
>><br>
>>> Mahesh,<br>
>>><br>
>>> run 'ip -s x s s spi <your_broken_inbound_spi' (as root) on your<br>
>>> Linux-Box and check if your error statistics increase for the particular :<br>
>>> <snip><br>
>>> stats:<br>
>>> replay-window 0 replay 0 failed 0<br>
>>> <snap><br>
>>> Also: 'grep -vw 0 /proc/net/xfrm_stat' and check for increasing<br>
>>> counters. You will probably have to rebuild your Linux-kernel for this,<br>
>>> unless it has the CONFIG_XFRM_STATISTICS option enabled. If the file<br>
>>> does exist you're lucky, if not -- like on current Debian systems -- you<br>
>>> will have to recompile.<br>
>>> The rationale behind this is that your inbound traffic gets dropped<br>
>>> during inbound transformation. Reasons for this may vary: failed<br>
>>> integrity checks, replay problems, failed inbound policy check etc.<br>
>>><br>
>>> Cheers,<br>
>>> Thomas<br>
>>><br>
>>><br>
>>> On 12/13/2015 05:06 PM, Mahesh Neelakanta wrote:<br>
>>>> Hi,<br>
>>>> I have a Strongswan VPN server that is being used to terminate VPN<br>
>>>> connections with multiple endpoints. Most of the existing endpoints are<br>
>>>> cisco, sophos, etc. Recently I have a Juniper ISG 1000 endpoint that is<br>
>>>> posing some intermittent traffic problems.<br>
>>>><br>
>>>> The exact issue is that traffic over the VPN pauses after some (random)<br>
>>>> time. The tunnel itself is up and at the next rekey traffic starts<br>
>>> flowing<br>
>>>> again. If I reduce the re-key time from 3600s down to 600s, the problem<br>
>>> is<br>
>>>> reduced significantly. I did verify with the remote side that their<br>
>>> keylife<br>
>>>> is 3600s. We do not have DPD enabled. There is constant traffic so there<br>
>>>> are no periods of inactivity.<br>
>>>><br>
>>>> During the periods where traffic pauses, ipsec statusall report shows no<br>
>>>> more packets in bytes_i (whereas bytes_o is still increasing).<br>
>>>><br>
>>>> Here is the config on our end (IPs and subnets have been changed for<br>
>>>> security):<br>
>>>><br>
>>>> config setup<br>
>>>> uniqueids = no<br>
>>>> charondebug = ike 2<br>
>>>><br>
>>>> conn %default<br>
>>>> keyingtries=%forever<br>
>>>> dpdaction=none<br>
>>>><br>
>>>> conn vpn-juniper-prd<br>
>>>> left=%defaultroute<br>
>>>> leftid=42.75.5.14 # Our actual local IP is 10.20.1.18, we are<br>
>>>> NATed going out<br>
>>>> leftsubnet=<a href="http://5.22.11.21/32" rel="noreferrer" target="_blank">5.22.11.21/32</a><br>
>>>> right=168.42.68.5<br>
>>>> rightid=168.42.68.5<br>
>>>> rightsubnet=<a href="http://12.23.0.0/16" rel="noreferrer" target="_blank">12.23.0.0/16</a><br>
>>>> keyexchange=ikev1<br>
>>>> ikelifetime=28800s<br>
>>>> ike=aes128-sha1-modp1024<br>
>>>> esp=aes128-sha1-modp1024<br>
>>>> keylife=3600m<br>
>>>> type=tunnel<br>
>>>> compress=no<br>
>>>> authby=secret<br>
>>>> auto=start<br>
>>>><br>
>>>> Notice that the last "bytes_i" shows 145s ago (ipsec statusall output):<br>
>>>><br>
>>>> vpn-juniper-prd: %any...168.42.68.5 IKEv1<br>
>>>> vpn-juniper-prd: local: [42.75.5.14] uses pre-shared key<br>
>>> authentication<br>
>>>> vpn-juniper-prd: remote: [168.42.68.5] uses pre-shared key<br>
>>> authentication<br>
>>>> vpn-juniper-prd: child: <a href="http://5.22.11.21/32" rel="noreferrer" target="_blank">5.22.11.21/32</a> === <a href="http://12.23.0.0/16" rel="noreferrer" target="_blank">12.23.0.0/16</a> TUNNEL<br>
>>>> vpn-juniper-prd[1]: ESTABLISHED 110 minutes ago,<br>
>>>> 10.20.1.18[42.75.5.14]...168.42.68.5[168.42.68.5]<br>
>>>> vpn-juniper-prd[1]: IKEv1 SPIs: a8ed9dd3b567a578_i* 97dbd6dbb3683aa4_r,<br>
>>>> pre-shared key reauthentication in 5 hours<br>
>>>> vpn-juniper-prd[1]: IKE proposal:<br>
>>>> AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024<br>
>>>> vpn-juniper-prd{44}: REKEYED, TUNNEL, reqid 4, expires in 10 minutes<br>
>>>> vpn-juniper-prd{44}: <a href="http://5.22.11.21/32" rel="noreferrer" target="_blank">5.22.11.21/32</a> === <a href="http://12.23.0.0/16" rel="noreferrer" target="_blank">12.23.0.0/16</a><br>
>>>> vpn-juniper-prd{52}: INSTALLED, TUNNEL, reqid 4, ESP SPIs: c3fdc693_i<br>
>>>> 9d90fe7f_o<br>
>>>> vpn-juniper-prd{52}: AES_CBC_128/HMAC_SHA1_96, 24197112 bytes_i *(26366<br>
>>>> pkts, 145s ago*), 8889197 bytes_o (31780 pkts, 0s ago), rekeying in 10<br>
>>>> minutes<br>
>>>> vpn-juniper-prd{52}: <a href="http://5.22.11.21/32" rel="noreferrer" target="_blank">5.22.11.21/32</a> === <a href="http://12.23.0.0/16" rel="noreferrer" target="_blank">12.23.0.0/16</a><br>
>>>><br>
>>>> During that time, we still see packets going in/out via the eth0<br>
>>> interface :<br>
>>>><br>
>>>> 03:38:52.349565 IP 10.20.1.18 > <a href="http://168.42.68.5" rel="noreferrer" target="_blank">168.42.68.5</a>:<br>
>>>> ESP(spi=0x9d90fe7f,seq=0x7c0f), length 132<br>
>>>> 03:38:52.363916 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd3), length 132<br>
>>>> 03:38:52.548261 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd4), length 100<br>
>>>> 03:38:52.564198 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd5), length 100<br>
>>>> 03:38:53.357693 IP 10.20.1.18 > <a href="http://168.42.68.5" rel="noreferrer" target="_blank">168.42.68.5</a>:<br>
>>>> ESP(spi=0x9d90fe7f,seq=0x7c10), length 132<br>
>>>> 03:38:53.371666 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd6), length 132<br>
>>>> 03:38:54.365616 IP 10.20.1.18 > <a href="http://168.42.68.5" rel="noreferrer" target="_blank">168.42.68.5</a>:<br>
>>>> ESP(spi=0x9d90fe7f,seq=0x7c11), length 132<br>
>>>> 03:38:54.379533 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd7), length 132<br>
>>>> 03:38:55.250707 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd8), length 100<br>
>>>> 03:38:55.373593 IP 10.20.1.18 > <a href="http://168.42.68.5" rel="noreferrer" target="_blank">168.42.68.5</a>:<br>
>>>> ESP(spi=0x9d90fe7f,seq=0x7c12), length 132<br>
>>>> 03:38:55.387695 IP 168.42.68.5 > <a href="http://10.20.1.18" rel="noreferrer" target="_blank">10.20.1.18</a>:<br>
>>>> ESP(spi=0xc3fdc693,seq=0x5cd9), length 132<br>
>>>><br>
>>>><br>
>>>> thanks,<br>
>>>> mahesh<br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Users mailing list<br>
>>>> <a href="mailto:Users@lists.strongswan.org">Users@lists.strongswan.org</a><br>
>>>> <a href="https://lists.strongswan.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.strongswan.org/mailman/listinfo/users</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Users mailing list<br>
>>> <a href="mailto:Users@lists.strongswan.org">Users@lists.strongswan.org</a><br>
>>> <a href="https://lists.strongswan.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.strongswan.org/mailman/listinfo/users</a><br>
>>><br>
>><br>
>><br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>