[strongSwan] Throughput on high BDP networks

jsullivan at opensourcedevel.com jsullivan at opensourcedevel.com
Wed Jun 3 04:23:23 CEST 2015


> On June 1, 2015 at 11:48 AM Martin Willi <martin at strongswan.org> wrote:
>
>
>
> > Even at these rates, the CPU did not appear to be very busy. We had one at
> > 85%
> > occupied but that was the one running nuttcp.
>
> On the outgoing path, the Linux kernel usually accounts ESP encryption
> under the process that sends traffic using a socket send() call. So
> these 85% probably include AES-GCM.
>
> On the receiving or forwarding path, you'll have to look at the software
> interrupt usage (si in top).
>
> > We have seen these boxes pass almost 20 Gbps with single digit
> > utilization so they have plenty of horsepower.
>
> That does not have to mean much. Its all about encryption, and that is
> rather expensive. If you have specialized hardware, this most likely
> means it is good at shuffling data over the network, but might be
> underpowered when it has to do encryption in software.
>
> > We are also running haveged on them to prevent entropy starvation for the
> > encryption.
>
> Only the key exchange needs entropy, raw AES-GCM does not.
>
> Regards
> Martin
>
 
Hello, all.  Still battling this problem.  The system is a SuperMicro server and
not a specialized device.  It looks like the problem may be software interrupts
but on the sending side so I am very curious about the recommendation to check
the receiving side.  In sending only 100 Mbps or so, I see a single CPU pegged
at 100% for si.  I wondered if it might be the number of ACK packets being
returned from the other side.  We have a large window size so we get a flood of
ACK packets in reply send but that really doesn't seem to make sense.  But it
would explain why we see this more in TCP than UDP . . . . or so we thought.  I
then sent 200 Mbps of UDP traffic so virtually no reply traffic and the sender
was still at 100% si.  What might be generating such a huge number of software
interrupts and how can we reduce them or spread them across multiple processors?

Here is an interesting screen scrape showing the high si utilization, the
multiple kworker threads from pcrypt and that the ethernet drivers are utilizing
multiple cores (at least I think that is what ethtool -L is showing).  Thanks -
John

top - 22:19:45 up 2 days, 18:49,  1 user,  load average: 0.88, 0.73, 0.47
Tasks: 161 total,   4 running, 157 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  1.8 sy,  0.0 ni, 36.6 id,  0.0 wa,  0.0 hi, 61.6 si,  0.0 st
%Cpu1  :  0.0 us,  4.7 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.7 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  1.0 sy,  0.0 ni, 98.7 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  1.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.7 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.7 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  1.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  :  0.0 us,  1.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32987140 total,  1931224 used, 31055916 free,   141196 buffers
KiB Swap:  7843824 total,        0 used,  7843824 free,  1110780 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
                                                                                 
21654 root      20   0     0    0    0 R    23  0.0   1:15.43 kworker/0:0
                                                                             
21695 root      20   0     0    0    0 S    19  0.0   0:01.68 kworker/0:2
                                                                             
20784 root      20   0     0    0    0 S    16  0.0   3:20.98 kworker/0:1
                                                                             
21668 root      20   0     0    0    0 R     3  0.0   0:52.09 kworker/0:3
                                                                             
21669 root      20   0     0    0    0 S     3  0.0   0:09.51 kworker/1:2
                                                                             
21560 root      20   0     0    0    0 S     2  0.0   0:22.76 kworker/1:1
                                                                             
 6605 root       0 -20 18508 6508  484 S     2  0.0  50:19.27 conntrackd
                                                                              
21613 root      20   0     0    0    0 S     1  0.0   0:07.16 kworker/8:2
                                                                             
21616 root      20   0     0    0    0 S     1  0.0   0:02.07 kworker/2:1
                                                                             
21621 root      20   0     0    0    0 S     1  0.0   0:01.18 kworker/9:0
                                                                             
21646 root      20   0     0    0    0 S     1  0.0   0:05.64 kworker/5:2
                                                                             
21653 root      20   0     0    0    0 R     1  0.0   0:05.48 kworker/3:0
                                                                             
21465 root      20   0     0    0    0 S     1  0.0   0:05.54 kworker/10:0
                                                                            
21611 root      20   0     0    0    0 S     1  0.0   0:05.18 kworker/6:0
                                                                             
21617 root      20   0     0    0    0 S     1  0.0   0:06.43 kworker/4:0
                                                                             
21618 root      20   0     0    0    0 S     1  0.0   0:01.76 kworker/11:1
                                                                            
21634 root      20   0     0    0    0 S     1  0.0   0:03.82 kworker/7:0
                                                                             
  362 root      20   0     0    0    0 S     0  0.0   0:06.10 jbd2/md0-8
                                                                              
 4460 quagga    20   0 26296 1396  608 S     0  0.0   0:05.52 zebra
                                                                                   
 5114 root      20   0 24372 4224 2088 S     0  0.0  37:38.64 openvpn
                                                                                 
21498 root      20   0     0    0    0 S     0  0.0   0:07.12 kworker/1:0
                                                                             
    1 root      20   0 10652  820  680 S     0  0.0   0:04.25 init
                                                                                    
    2 root      20   0     0    0    0 S     0  0.0   0:00.06 kthreadd
                                                                                
    3 root      20   0     0    0    0 S     0  0.0   6:56.21 ksoftirqd/0
                                                                             
    6 root      rt   0     0    0    0 S     0  0.0   0:05.71 migration/0
                                                                             
    7 root      rt   0     0    0    0 S     0  0.0   0:22.36 watchdog/0
                                                                              
    8 root      rt   0     0    0    0 S     0  0.0   0:00.45 migration/1
                                                                             
   10 root      20   0     0    0    0 S     0  0.0   0:00.32 ksoftirqd/1
                                                                             
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth^C
root at gwhq-1:/etc/ipsec.d/remotenets# ip link ls | grep bond1
7: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1
state UP mode DEFAULT group default qlen 1000
10: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1
state UP mode DEFAULT group default qlen 1000
15: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP mode DEFAULT group default 
32: vrrp.1 at bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state
UNKNOWN mode DEFAULT group default 
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth3
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1combined 8
root at gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth5
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1combined 8


More information about the Users mailing list