[strongSwan] Using multiple UDP sockets with SO_REUSEPORT option to increase high connection rate

[Chinmaya Dwibedy]

Hi Martin,

Thanks a lot Martin for your prompt response and valuable
suggestion as well. 
I have configured the number of threads to 32 at both
the ends (IKE responder and IKE Initiator). At IKE Initiator end, if I increase
the sender threads (i.e., initiators in load-tester section) from 5 to 10
(i.e., to put load on all cores), I find the followings at IKE Responder end.

	1. There are huge packets loss (i.e., packet
receive errors in the Udp section) in #netstat –su.
	2. The Recv-Q column of the isakmp connection
is high and doesn't drop to zero in  #netstat –ua.

From the above statistics, it just means that Receiver
thread ( of Charon daemon at IKE Responder end) is not reading the socket fast
enough. The average arrival rate regularly causes a backlog in the receive
queue. The maximum number of queued received data depends on
/proc/sys/net/ipv4/udp_mem and /proc/sys/net/ipv4/udp_rmem_min. Thus I increased
these values to 8MB before the start of test but still faced the same issue.
Note that, if I keep the initiators and iterations to 5 and 50000 respectively,
then there was no loss at all and I am getting the setup rate to be
220-225  

Here goes the output of “ipsec statusall" at IKE
responder end during test?

Status of IKE charon daemon (strongSwan 5.0.4, Linux
2.6.34.10-grsec-BenuOcteon,

 mips64):

  uptime: 2
minutes, since Jan 01 00:11:15 1970

  malloc: sbrk
500166656, mmap 5005312, used 500018224, free 148432

  worker threads:
24 of 32 idle, 6/2/0/0 working, job queue: 0/2/0/0, scheduled:

 37639

  loaded plugins:
charon aes des sha1 sha2 md5 random nonce x509 revocation cons

traints pubkey pkcs1 pkcs8 pgp dnskey pem fips-prf gmp
xcbc cmac hmac attr kerne

l-netlink resolve socket-default stroke updown
xauth-generic

Virtual IP pools (size/online/offline):

  10.0.0.0/8:
16777214/21757/0


The job queue load shows 2 jobs are queued for CRITICAL
priority. I also profiled the Charon daemon (IKE Responder) and found that,
most of the threads are getting blocked in pthread_cond_wait ().   It means the thread currently has no work to
do and waits for a job. Any suggestion what limits this connection rate or any pointer
on how to debug this issue further?

Regards,

Chinmaya





On Friday, March 21, 2014 8:23 PM, Martin Willi <martin at strongswan.org> wrote:
  

> And the single receiver thread becomes bottleneck due to high
> connection rate/setup rate.

The receiver job is rather trivial, only the IKE header is parsed and
some rate limiting is enforced for DoS protection. Any further
processing is delegated to the thread pool using that
process_message_job(). So I have my doubts that this is the bottleneck
you are actually looking for.

> Can it possible to create separate UDP sockets for each thread? The
> SO_REUSEPORT socket option allows multiple UDP sockets to be bound to
> the same port. With SO_REUSEPORT, multiple threads could use recvfrom
> () on its own socket to accept datagrams arriving on the port.

Theoretically that is possible, but I'm not really sure if that helps to
fix the issues you are seeing.

When running your tests, how does your job queue in "ipsec statusall"
look like? If you have many jobs queued that don't get processed,
something else prevents that charon scales properly on your platform.


Regards
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20140324/bc3b4c5d/attachment.html>