[strongSwan] Soaking GigE bidirectionally

[Robert Woodcock]

This may very well not be a strongSwan issue per se, but I'm hoping that one of you may be able to steer me towards the right path.

I am trying to build a VPN router that will not be a bottleneck for a GigE connection. Ideally, it would also have quite a bit of headroom for smaller packets (the average in our environment is about 900 bytes, since a fair bit of our traffic is interactive in nature), a few dozen firewall rules, some dynamic routing, and traffic accounting. So far it looks promising, but I think I'm running into problems getting all of the important things multithreaded.

I've been benchmarking this on two identical systems connected through an ethernet crossover cable (standard 1500 byte MTU), and on top of that an IPSec ESP tunnel using aes-sha1. The hardware is as follows:
Intel Xeon E3-1270 (3.5GHz, 4 cores, hyperthreading enabled)
Intel S1200BTL server board
Intel 82571EB ethernet cards

Software is currently:
Debian 6.0 x64
Linux 3.2.23 (SMP enabled, pcrypt and aesni_intel modules loaded, e1000e driver)
strongSwan 4.4.1 (as comes with Debian 6.0)
iperf and nuttcp for bandwidth testing (nuttcp seems to take a lot less CPU)

I am a very recent convert from openSwan - my first few benchmarking attempts showed that its lack of multithreading was a bottleneck, and I was only getting about 540Mbit throughput across a tunnel, so I switched to strongSwan. In fact I'm currently using Linux 3.2.23 simply because that's the last version the openSwan KLIPS patches currently apply against (although I'm of course using NETKEY for these strongSwan tests).

With strongSwan, soaking gigabit one direction is no problem (I consistently get results of about 902Mbit/sec over the tunnel vs 941Mbit/sec over the ethernet connection, and the discrepancy nearly exactly matches the ESP encapsulation overhead). However, when I use iperf's -d option to test bidirectionally, or when I manually run nuttcp from both hosts at once, one side or the other will be quite starved for throughput, such that the total send+receive bandwidth is about 1000-1300Mbit/sec, and one side may get less than 100Mbit/sec through. More interestingly, though, I show this same starved IPSec throughput when the other direction is soaked with non-IPsec traffic! This is with both the 1.5.1k e1000e driver that comes with 3.2.23 as well as the most recent 2.0.0.1-NAPI driver that Intel has available for download. I'm also not seeing any difference in speeds with the pcrypt module loaded vs. unloaded.
 
In these examples, router A's ethernet IP is 192.168.199.1, B's is 192.168.199.2, and the VPN tunnel IP (on interface dummy0) for A is 172.30.199.1 and B is 172.30.199.2. I repeated each test three times. CPU usage (measured via top) on both hosts for the Ethernet-only tests are about 2.5% - for the IPSec tests and IPSec/Ethernet tests are 9-10%.

***** Ethernet only *****
RouterA# nuttcp 192.168.199.2
 1120.6133 MB /  10.02 sec =  937.8221 Mbps 1 %TX 7 %RX 0 retrans 0.21 msRTT
RouterB# nuttcp 192.168.199.1
  864.0843 MB /  10.03 sec =  722.4173 Mbps 1 %TX 5 %RX 0 retrans 13.10 msRTT

RouterA# nuttcp 192.168.199.2
 1122.7757 MB /  10.03 sec =  938.9298 Mbps 1 %TX 7 %RX 0 retrans 0.26 msRTT
RouterB# nuttcp 192.168.199.1
  899.4255 MB /  10.03 sec =  752.4329 Mbps 1 %TX 6 %RX 0 retrans 7.60 msRTT

RouterA# nuttcp 192.168.199.2
  912.5861 MB /  10.02 sec =  763.7507 Mbps 1 %TX 6 %RX 0 retrans 7.52 msRTT
RouterB# nuttcp 192.168.199.1
 1122.3195 MB /  10.06 sec =  936.1915 Mbps 1 %TX 7 %RX 0 retrans 0.25 msRTT

***** IPSec both ways *****
RouterA# nuttcp 172.30.199.2
  838.1395 MB /  10.01 sec =  702.3926 Mbps 22 %TX 26 %RX 0 retrans 0.32 msRTT
RouterB# nuttcp 172.30.199.1
  519.9860 MB /  10.05 sec =  434.0403 Mbps 9 %TX 21 %RX 0 retrans 2.28 msRTT

RouterA# nuttcp 172.30.199.2
 1070.0625 MB /  10.00 sec =  897.3043 Mbps 38 %TX 19 %RX 0 retrans 0.39 msRTT
RouterB# nuttcp 172.30.199.1
  115.9592 MB /  10.00 sec =   97.2437 Mbps 2 %TX 4 %RX 0 retrans 0.37 msRTT

RouterA# nuttcp 172.30.199.2
 1072.0000 MB /  10.01 sec =  898.4926 Mbps 35 %TX 15 %RX 0 retrans 0.37 msRTT
RouterB# nuttcp 172.30.199.1
  106.1074 MB /  10.00 sec =   88.9860 Mbps 4 %TX 1 %RX 0 retrans 0.53 msRTT

***** IPSec one way, ethernet the other way *****
RouterA# nuttcp 172.30.199.2
  522.4740 MB /  10.05 sec =  436.0902 Mbps 23 %TX 6 %RX 0 retrans 0.39 msRTT
RouterB# nuttcp 192.168.199.1
 1048.9179 MB /  10.03 sec =  877.3937 Mbps 1 %TX 9 %RX 0 retrans 8.53 msRTT

RouterA# nuttcp 172.30.199.2
  119.3661 MB /  10.05 sec =   99.6532 Mbps 4 %TX 1 %RX 0 retrans 0.38 msRTT
RouterB# nuttcp 192.168.199.1
 1118.0277 MB /  10.03 sec =  935.3272 Mbps 1 %TX 10 %RX 0 retrans 0.49 msRTT

RouterA# nuttcp 172.30.199.2
  225.8911 MB /  10.04 sec =  188.7413 Mbps 8 %TX 3 %RX 0 retrans 0.40 msRTT
RouterB# nuttcp 192.168.199.1
 1115.9659 MB /  10.04 sec =  932.7818 Mbps 1 %TX 12 %RX 0 retrans 0.24 msRTT