[strongSwan] Throughput on high BDP networks
jsullivan at opensourcedevel.com
jsullivan at opensourcedevel.com
Wed Jun 3 04:23:23 CEST 2015
> On June 1, 2015 at 11:48 AM Martin Willi <martin at strongswan.org> wrote:
>
>
>
> > Even at these rates, the CPU did not appear to be very busy. We had one at
> > 85%
> > occupied but that was the one running nuttcp.
>
> On the outgoing path, the Linux kernel usually accounts ESP encryption
> under the process that sends traffic using a socket send() call. So
> these 85% probably include AES-GCM.
>
> On the receiving or forwarding path, you'll have to look at the software
> interrupt usage (si in top).
>
> > We have seen these boxes pass almost 20 Gbps with single digit
> > utilization so they have plenty of horsepower.
>
> That does not have to mean much. Its all about encryption, and that is
> rather expensive. If you have specialized hardware, this most likely
> means it is good at shuffling data over the network, but might be
> underpowered when it has to do encryption in software.
>
> > We are also running haveged on them to prevent entropy starvation for the
> > encryption.
>
> Only the key exchange needs entropy, raw AES-GCM does not.
>
> Regards
> Martin
>
Hello, all. Still battling this problem. The system is a SuperMicro server and
not a specialized device. It looks like the problem may be software interrupts
but on the sending side so I am very curious about the recommendation to check
the receiving side. In sending only 100 Mbps or so, I see a single CPU pegged
at 100% for si. I wondered if it might be the number of ACK packets being
returned from the other side. We have a large window size so we get a flood of
ACK packets in reply send but that really doesn't seem to make sense. But it
would explain why we see this more in TCP than UDP . . . . or so we thought. I
then sent 200 Mbps of UDP traffic so virtually no reply traffic and the sender
was still at 100% si. What might be generating such a huge number of software
interrupts and how can we reduce them or spread them across multiple processors?
Here is an interesting screen scrape showing the high si utilization, the
multiple kworker threads from pcrypt and that the ethernet drivers are utilizing
multiple cores (at least I think that is what ethtool -L is showing). Thanks -
John
top - 22:19:45 up 2 days, 18:49, 1 user, load average: 0.88, 0.73, 0.47
Tasks: 161 total, 4 running, 157 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 1.8 sy, 0.0 ni, 36.6 id, 0.0 wa, 0.0 hi, 61.6 si, 0.0 st
%Cpu1 : 0.0 us, 4.7 sy, 0.0 ni, 95.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 1.0 sy, 0.0 ni, 98.7 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32987140 total, 1931224 used, 31055916 free, 141196 buffers
KiB Swap: 7843824 total, 0 used, 7843824 free, 1110780 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21654 root 20 0 0 0 0 R 23 0.0 1:15.43 kworker/0:0
21695 root 20 0 0 0 0 S 19 0.0 0:01.68 kworker/0:2
20784 root 20 0 0 0 0 S 16 0.0 3:20.98 kworker/0:1
21668 root 20 0 0 0 0 R 3 0.0 0:52.09 kworker/0:3
21669 root 20 0 0 0 0 S 3 0.0 0:09.51 kworker/1:2
21560 root 20 0 0 0 0 S 2 0.0 0:22.76 kworker/1:1
6605 root 0 -20 18508 6508 484 S 2 0.0 50:19.27 conntrackd
21613 root 20 0 0 0 0 S 1 0.0 0:07.16 kworker/8:2
21616 root 20 0 0 0 0 S 1 0.0 0:02.07 kworker/2:1
21621 root 20 0 0 0 0 S 1 0.0 0:01.18 kworker/9:0
21646 root 20 0 0 0 0 S 1 0.0 0:05.64 kworker/5:2
21653 root 20 0 0 0 0 R 1 0.0 0:05.48 kworker/3:0
21465 root 20 0 0 0 0 S 1 0.0 0:05.54 kworker/10:0
21611 root 20 0 0 0 0 S 1 0.0 0:05.18 kworker/6:0
21617 root 20 0 0 0 0 S 1 0.0 0:06.43 kworker/4:0
21618 root 20 0 0 0 0 S 1 0.0 0:01.76 kworker/11:1
21634 root 20 0 0 0 0 S 1 0.0 0:03.82 kworker/7:0
362 root 20 0 0 0 0 S 0 0.0 0:06.10 jbd2/md0-8
4460 quagga 20 0 26296 1396 608 S 0 0.0 0:05.52 zebra
5114 root 20 0 24372 4224 2088 S 0 0.0 37:38.64 openvpn
21498 root 20 0 0 0 0 S 0 0.0 0:07.12 kworker/1:0
1 root 20 0 10652 820 680 S 0 0.0 0:04.25 init
2 root 20 0 0 0 0 S 0 0.0 0:00.06 kthreadd
3 root 20 0 0 0 0 S 0 0.0 6:56.21 ksoftirqd/0
6 root rt 0 0 0 0 S 0 0.0 0:05.71 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:22.36 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.45 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/1
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth^C
root at gwhq-1:/etc/ipsec.d/remotenets# ip link ls | grep bond1
7: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1
state UP mode DEFAULT group default qlen 1000
10: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1
state UP mode DEFAULT group default qlen 1000
15: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP mode DEFAULT group default
32: vrrp.1 at bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state
UNKNOWN mode DEFAULT group default
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth3
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1combined 8
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth5
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1combined 8
More information about the Users
mailing list