[strongSwan] Issue with huge packets loss at IKE Responder under high load

[Chinmaya Dwibedy]

Hi All,
I am doing scalability/load test (250k IPsec tunnels) using
load tester plugin strong swan 5.0.4).
I have configured the number of threads to 32 at both the ends
(IKE responder and IKE Initiator). At IKE Initiator end, if I increase the
sender threads (i.e., initiators in load-tester section) from 5 to 10 (i.e., to
put load on all cores), I find the followings at IKE Responder end.
	1. There are huge packets loss (i.e., packet receive
errors in the Udp section) in #netstat –su.
	2. The Recv-Q column of the isakmp connection is high
and doesn't drop to zero in  #netstat
–ua.

Note that, if I keep the initiators and iterations to 5 and 50000 respectively,
then there was no loss at all.  
The Receiver listens on the UDP socket for incoming IKE
messages. If it receives an IKE message over the Socket he creates a
“process_message_job” for the message and adds this job to the Jobqueue of the
Processor. The IKE message is then processed through the Processor and its
threads. The Receiver has no separate thread. There is just a “receiver” job
which is executed regularly by the Processor.
The Receiver that picks up data from the socket buffers and
hands it over for processing is unable to keep up. The average arrival rate
regularly causes a backlog in the receive queue. I also increased the maximum
number of queued received data via /proc/sys/net/ipv4/udp_mem and
/proc/sys/net/ipv4/udp_rmem_min, but it could not prevent loss.
I also profiled the Charon daemon (IKE Responder) and found
that, most of the threads are getting blocked in pthread_cond_wait().   It
means the thread currently has no work to do and waits for a job. 
Can anyone please suggest what might be the issues behind this
slow receiver (at IKE Responder end) and the ways to resolve the same?
Your help in this regard will be highly appreciated.  
 
Regards,
Chinmaya 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20140320/825dca09/attachment.html>