[strongSwan] Performance issue with 20k IPsec tunnels (using 5.0.4 strongswan and load-tester plugin)

[Chinmaya Dwibedy]

Hi Martin,
Thanks a lot for your valuable
feedback. I modified the strongswan configuration to run 10 threads instead of
100 threads also other parameters. I could able to scale up to 20k ipsec
tunnels without traffic. While trying with 25K IPsec connections, it could
bring up 24855 IPsec connections at both ends (IKE initiator and IKE
responder). In order to debug this issue, I used load-tester command line tool (ipsec
load-tester initiate 25000 30) as mentioned at http://wiki.strongswan.org/projects/strongswan/wiki/LoadTests.
I find, there are lots of retransmissions (as it prints the status of the
initiation with *character mostly) in console. I know, these are certainly considered
to be bad. But I have set the retransmit_timeout and retransmit_tries to 300
seconds and 300 times respectively, which is a huge. Also checked the CPU usage
of Charon daemon at IKE responder end (through top –p <PID of Charon daemon>)
and found to be less than 20% (mostly). Can you please guide/suggest what might be the issue?
Or profiling to find out the potential bottleneck is the only option left. Few
days ago, I had read in this mailing chain that, handling 70K tunnels should be
doable on a single box if we have high end hardware. 
 
Here goes the configuration
 IKE Responder
Strongswan.conf
# number of worker threads in
charon
        threads = 32
        replay_window = 32
        lock_threshold=35000
        cookie_threshold=35000
        init_limit_half_open=35000
        half_open_timeout=35000
        dos_protection = no
        close_ike_on_child_failure=yes
        ikesa_table_size = 8192
        ikesa_table_segments = 32
        reuse_ikesa = no
 
 
   ipsec.conf
conn %default
        ikelifetime=24h
        keylife=23h
        rekeymargin=5m
        keyingtries=1
        keyexchange=ikev2
        ike=aes128-sha1-modp1024!
        mobike=no
 
conn gw-gw
        left=30.30.30.21
        leftsubnet=40.0.0.1/8
        rightid=%any
        leftauth=psk
        leftfirewall=yes
        rightsourceip=10.0.0.0/8
        leftid=@srv.strongswan.org
        rightauth=psk
        type=tunnel
        authby=secret
        rekey=no
        reauth=no
        auto=add
 
ipsec.secrets
@srv.strongswan.org %any : PSK
"strongSwan"
 
IKE Initiator 
Strongswa.conf
        threads = 32
        replay_window = 32
        dos_protection = no
        block_threshold=35000
        cookie_threshold=35000
        init_limit_half_open=35000
        retransmit_timeout=300
        retransmit_tries=300
        install_virtual_ip=no
        install_routes=no
        close_ike_on_child_failure=yes
        ikesa_table_size = 8192
        ikesa_table_segments = 32
        reuse_ikesa = no
 
load-tester {
                   enable = yes
                   initiators = 10
                   iterations = 2500
                   delay = 20
                   responder = 30.30.30.21
                   initiator_tsr=40.0.0.1/8
                   proposal =
aes128-sha1-modp1024
                   initiator_auth = psk
                   responder_auth = psk
                   request_virtual_ip = yes
                   ike_rekey = 0
                   child_rekey = 0
                   delete_after_established =
no
                   shutdown_when_complete = no
 
                  }
 
ipsec.secrets
@srv.strongswan.org %any : PSK
"strongSwan"
 
Regards,
Chinmaya
 

________________________________
 From: Martin Willi <martin at strongswan.org>
To: Chinmaya Dwibedy <ckdwibedy at yahoo.com> 
Cc: "users at lists.strongswan.org" <users at lists.strongswan.org> 
Sent: Thursday, September 19, 2013 4:56 PM
Subject: Re: [strongSwan] Performance issue with 20k IPsec tunnels (using 5.0.4 strongswan and load-tester plugin)
  

Hi,

>    threads = 32
>    load-tester {
>        initiators = 100

That won't work. As you can read on [1], each initiator is a thread
creating connections. But you have much more initiators configured than
your pool has threads. Likely that your threads all are busy initiating,
but none is processing incoming packets.

Running 100 initiators makes hardly sense. Usually you might need a few
to put load on all your cores for the DH exchange, but more than 10 are
usually not needed.

To find the bottleneck of your setup, you'll have to do some profiling.
First you'll have to check if the initiator or the responder hits some
limits. Use a tool of your choice.

It also might help to use the load-tester command line tool. It gives
you valuable feedback; retransmissions are bad and mean that you hit a
limit either on the initiator or the responder.

Regards
Martin

[1]http://wiki.strongswan.org/projects/strongswan/wiki/LoadTests
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20130924/e06eb476/attachment.html>