[strongSwan] Performance (latency) in a Hub and Spoke setup

Noel Kuntze noel.kuntze+strongswan-users-ml at thermi.consulting
Sat Jan 20 14:29:36 CET 2018 

Hi,

Capture it all those locations and compare.
You probably want TCP port 80/443 and ICMP.

Kind regards

Noel

On 20.01.2018 09:51, Martin Sand wrote:
>
> I am at the other location right now.
>
> Where should I capture the traffic - hub, spoke router, spoke http server, my client?
>
> Best regards
> Martin
>
>
> On 01/03/2018 11:17 PM, Noel Kuntze wrote:
>> Hi,
>>
>> If you used tracepath -T, then that message you posted earlier could indeed be caused by tracepath and not be the actual problem.
>>
>> Did you actually test that? What is the upload speed of the router? I strongly doubt the problem with the HTTP latency is caused by a throughput problem.
>> Could you possibly provide a tcpdump of traffic when the problem occurs?
>>
>> Kind regards
>>
>> Noel
>>
>> On 01.01.2018 16:21, Martin Sand wrote:
>>> Thanks Noel and Thomas.
>>>
>>> I did a lot of investigation over the weekend and it seems like these error messages are traceroute and tracepath specific issues.
>>> There was a post on serverfault explaining the background [1]. So I will not further invest into this.
>>>
>>> So I think I cannot further improve the performance. It is limited by the upload speed of the spoke routers.
>>>
>>> Happy New Year and best regards
>>> Martin
>>>
>>>
>>> [1] https://serverfault.com/questions/623996/how-to-enable-traceroute-in-linux-machine
>>>
>>>
>>>
>>> On 30.12.2017 23:03, Noel Kuntze wrote:
>>>> Hi Martin,
>>>>
>>>> That can be relevant.
>>>>
>>>> That is an ICMP message of the router or recipient 210.211.212.213 to 192.168.2.135 complaining that the TTL [ of the TCP packet from 192.168.2.135 to 192.168.1.130 with the ID 63979 ] reached 0. Under the strong assumption
>>>> that a standard TTL is used (meaning you didn't change it to some low value), that means that you have a routing loop somewhere in your network, that the complained about packet got into.
>>>>
>>>> TL;DR: You likely got a routing loop. You need to find and fix it.
>>>>
>>>> Kind regards
>>>>
>>>> Noel
>>>>
>>>> On 30.12.2017 22:47, Martin Sand wrote:
>>>>> Hi Noel
>>>>>
>>>>> Thanks for the advice. I installed tcpdump and wireshark and added a rule to log ICMP errors.
>>>>> This is an excerpt from the log file. I assume this line shows something is sent to port 80 but I cannot find the corresponding iptables entry.
>>>>>
>>>>> Dec 30 21:42:11 localhost kernel: [1423944.393321] IN= OUT=eth0 SRC=210.211.212.213 DST=192.168.2.135 LEN=88 TOS=0x00 PREC=0xC0 TTL=64 ID=38805 PROTO=ICMP TYPE=11 CODE=0 [SRC=192.168.2.135 DST=192.168.1.130 LEN=60 TOS=0x00 PREC=0x00 TTL=1 ID=63979 DF PROTO=TCP SPT=47511 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 ]
>>>>>
>>>>> Best regards
>>>>> Martin
>>>>>
>>>>>
>>>>> On 28.12.2017 01:43, Noel Kuntze wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Looks like your firewall rules on the hub are broken and cause the problems or you need to configure an additional CHILD_SA to tunnel ICMP errors from the hub, because it has no IP in the local TS.
>>>>>> Check both those suspicions.
>>>>>>
>>>>>> Kind regards
>>>>>>
>>>>>> Noel
>>>>>>
>>>>>> On 27.12.2017 23:00, Martin Sand wrote:
>>>>>>> Thanks again Noel.
>>>>>>>
>>>>>>> I have executed `traceroute -T --mtu <destination>` and `mtr -rw <destination>` on machines at both locations.
>>>>>>> I did not do further investigation on the MSS yet since I have this strange packet loss.
>>>>>>> Based on the route, I assume this happens at the hub which is in between the two routers?
>>>>>>> Could this be the root cause I need to further investigate?
>>>>>>>
>>>>>>> Kind regards
>>>>>>> Martin
>>>>>>>
>>>>>>> traceroute -T --mtu pi-frankfurt
>>>>>>> traceroute to pi-frankfurt (192.168.2.135), 30 hops max, 60 byte packets
>>>>>>>    1  router-freiburg (192.168.1.1)  0.263 ms  0.179 ms  0.172 ms
>>>>>>>    2  * * *
>>>>>>>    3  router-frankfurt (192.168.2.1)  41.762 ms  41.182 ms  36.716 ms
>>>>>>>    4  pi-frankfurt (192.168.2.135)  36.693 ms  43.629 ms  37.051 ms
>>>>>>>
>>>>>>> traceroute -T --mtu pi-freiburg
>>>>>>> traceroute to pi-freiburg (192.168.1.130), 30 hops max, 60 byte packets
>>>>>>>    1  router-frankfurt (192.168.2.1)  0.489 ms  0.381 ms  0.287 ms
>>>>>>>    2  * * *
>>>>>>>    3  router-freiburg (192.168.1.1)  38.368 ms  47.673 ms  35.441 ms
>>>>>>>    4  pi-freiburg (192.168.1.130)  39.456 ms  54.566 ms  36.117 ms
>>>>>>>
>>>>>>> mtr -rw pi-frankfurt
>>>>>>> Start: 2017-12-27T22:57:40+0100
>>>>>>> HOST: workstation              Loss%   Snt   Last   Avg  Best Wrst StDev
>>>>>>>     1.|-- router-freiburg         0.0%    10    0.2   0.2   0.2 0.3   0.0
>>>>>>>     2.|-- ???                      100.0    10    0.0   0.0   0.0 0.0   0.0
>>>>>>>     3.|-- router-frankfurt        0.0%    10   33.3  35.5  32.5 42.0   2.7
>>>>>>>     4.|-- pi-frankfurt              0.0%    10   33.5  34.4  32.7 36.7   1.5
>>>>>>>
>>>>>>>
>>>>>>> On 27.12.2017 21:08, Noel Kuntze wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> You can test the convergence speed using `traceroute -T --mtu <destination>`, but that only gives you the MTU. You need to manually discover the MSS
>>>>>>>> using `traceroute -T -O mss=<mss> <destination>`.
>>>>>>>>
>>>>>>>> The best way to check if the problem continues is to just run tcpdump/wireshark and check for ICMP Fragmenation needed packets and TCP errors or timeouts.
>>>>>>>>
>>>>>>>> Kind regards
>>>>>>>>
>>>>>>>> Noel
>>>>>>>>
>>>>>>>> On 27.12.2017 17:12, Martin Sand wrote:
>>>>>>>>> Thanks Noel. Sorry, I had to travel to the other location (350 km).
>>>>>>>>>
>>>>>>>>> I adapted the iptable rules. It improved, but I have the impression it only improved a bit.
>>>>>>>>> Is there a way to measure MTU discovery time?
>>>>>>>>>
>>>>>>>>> Kind regards
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 14.12.2017 13:51, Noel Kuntze wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>> VPN internal http requests to a web server of another spoke take some time until the page is rendered.
>>>>>>>>>>> I assume this is due to the latency.
>>>>>>>>>> Nah. It's extremely more likely that the path MTU discovery takes some time (maybe due to some missing/wrong firewall rules on some host(s) in your network topology).
>>>>>>>>>> Try lowering the MTU and MSS of the tunneled traffic[1].
>>>>>>>>>>
>>>>>>>>>> Kind regards
>>>>>>>>>>
>>>>>>>>>> Noel
>>>>>>>>>>
>>>>>>>>>> [1] https://wiki.strongswan.org/projects/strongswan/wiki/ForwardingAndSplitTunneling#MTUMSS-issues
>>>>>>>>>>
>>>>>>>>>> On 14.12.2017 13:41, Martin Sand wrote:
>>>>>>>>>>> Hi all
>>>>>>>>>>>
>>>>>>>>>>> I have a Hub and Spoke setup. Connections are working perfectly fine.
>>>>>>>>>>> Throughput is almost reaching the maximum rate of the upload channel speed, 10 MBit/s.
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately the latency is not fulfilling my objectives. I have an average ping time of 39 ms (see below) when pinging clients on other spokes.
>>>>>>>>>>> VPN internal http requests to a web server of another spoke take some time until the page is rendered.
>>>>>>>>>>> I assume this is due to the latency.
>>>>>>>>>>>
>>>>>>>>>>> Is there any chance to improve the latency? Or is the latency perfectly good?
>>>>>>>>>>>
>>>>>>>>>>> Best regards
>>>>>>>>>>> Martin
>>>>>>>>>>>
>>>>>>>>>>> Hub internet address
>>>>>>>>>>> 64 bytes from vpn.example.com (217.122.5.6): icmp_seq=1 ttl=57 time=15.2 ms
>>>>>>>>>>>
>>>>>>>>>>> Internal address of Hub
>>>>>>>>>>> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
>>>>>>>>>>> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=62 time=40.4 ms
>>>>>>>>>>>
>>>>>>>>>>> Client on another spoke
>>>>>>>>>>> PING 192.168.1.130 (192.168.1.130) 56(84) bytes of data.
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=1 ttl=61 time=108 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=2 ttl=61 time=41.8 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=3 ttl=61 time=38.0 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=4 ttl=61 time=35.2 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=5 ttl=61 time=36.4 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=6 ttl=61 time=39.1 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=7 ttl=61 time=38.1 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=8 ttl=61 time=41.6 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=9 ttl=61 time=36.0 ms
>>>>>>>>>>> 64 bytes from 192.168.1.130: icmp_seq=10 ttl=61 time=36.7 ms
>>>>>>>>>>>
>>>>>>>>>>> --- 192.168.1.130 ping statistics ---
>>>>>>>>>>> 10 packets transmitted, 10 received, 0% packet loss, time 9013ms
>>>>>>>>>>> rtt min/avg/max/mdev = 35.295/45.159/108.281/21.146 ms
>>>>>>>>>>>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.strongswan.org/pipermail/users/attachments/20180120/2916f409/attachment.sig>

More information about the Users mailing list