[strongSwan] Throughput on high BDP networks

[jsullivan at opensourcedevel.com]

Hello, all.  We are attempting to use StrongSWAN on a fast (1 Gbps CIR one side
and 4x10Gbps on the other) with about 80ms latency so pretty high bandwidth
delay product.  The traffic is GRE/IPSec.  Our benchmarks show we can saturate
the 1 Gbps side with just GRE sustaining high 800 low 900 Mbps.  When we
activate IPSec, we plummet to around 40 Mbps - maybe we'll hit 400 Mbps on
occasion.
 
This seems to be a TCP windowing problem provoked by TCP segment
retransmissions.  When we use nstat between runs, GRE shows virtually no segment
retransmissions where GRE/IPSec shows thousands.  GRE tunnel MTU is 1412 so it
should be fine for both transport and tunnel mode.
 
sanitized config is:

type=transport
esp=aes128gcm8-modp1024

leftprotoport=47
rightprotoport=47
dpddelay=9
dpdtimeout=30
compress=yes

keyingtries=20
keylife=60m
rekeymargin=5m
ikelifetime=3h
mobike=no

authby=rsasig
rightrsasigkey=%cert

nat_traversal=yes
charonstart=yes
plutostart=yes

 

We are using intel cards with igb on one side and ixgbe on the other.

What do we need to do to eliminate the lost packets and where can we see the
drops? I don't see them on any queues, qdiscs - no stats showing packet drops.
Thanks - John