<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12pt"><div><span><br></span></div><div><br></div><span><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Hi Martin,</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Thanks a lot Martin for your prompt response and valuable
suggestion as well. </span></div><div><font face="Times New Roman">
</font><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>I have configured the number of threads to 32 at both
the ends (IKE responder and IKE Initiator). At IKE Initiator end, if I increase
the sender threads (i.e., initiators in load-tester section) from 5 to 10
(i.e., to put load on all cores), I find the followings at IKE Responder end.</span></div><div><font face="Times New Roman">
<br></font></div><ol style="list-style-type: decimal; direction: ltr;"><li style="color: rgb(0, 0, 0); font-size: 10pt; font-style: normal; font-weight: normal;"><div style="color: rgb(0, 0, 0); font-size: 11pt; font-style: normal; font-weight: normal; margin-top: 0in; margin-bottom: 0pt; mso-add-space: auto; mso-list: l0 level1 lfo1;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>There are huge packets loss (i.e., packet
receive errors in the Udp section) in #netstat –su.</span></div></li><li style='color: rgb(0, 0, 0); font-family: "Verdana","sans-serif"; font-size: 10pt; font-style: normal; font-weight: normal;'><div style='color: rgb(0, 0, 0); font-family: "Calibri","sans-serif"; font-size: 11pt; font-style: normal; font-weight: normal; margin-top: 0in; margin-bottom: 10pt; mso-add-space: auto; mso-list: l0 level1 lfo1;'><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>The Recv-Q column of the isakmp connection
is high and doesn't drop to zero in<span style="mso-spacerun: yes;">
</span>#netstat –ua.</span></div></li></ol><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>From the above statistics, it just means that Receiver
thread ( of Charon daemon at IKE Responder end) is not reading the socket fast
enough. The average arrival rate regularly causes a backlog in the receive
queue. The maximum number of queued received data depends on
/proc/sys/net/ipv4/udp_mem and /proc/sys/net/ipv4/udp_rmem_min. Thus I increased
these values to 8MB before the start of test but still faced the same issue.
Note that, if I keep the initiators and iterations to 5 and 50000 respectively,
then there was no loss at all and I am getting the setup rate to be
220-225<span style="mso-spacerun: yes;"> </span></span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Here goes the output of “ipsec statusall" at IKE
responder end during test?</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Status of IKE charon daemon (strongSwan 5.0.4, Linux
2.6.34.10-grsec-BenuOcteon,</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>mips64):</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>uptime: 2
minutes, since Jan 01 00:11:15 1970</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>malloc: sbrk
500166656, mmap 5005312, used 500018224, free 148432</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>worker threads:
24 of 32 idle, 6/2/0/0 working, job queue: 0/2/0/0, scheduled:</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>37639</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>loaded plugins:
charon aes des sha1 sha2 md5 random nonce x509 revocation cons</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>traints pubkey pkcs1 pkcs8 pgp dnskey pem fips-prf gmp
xcbc cmac hmac attr kerne</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>l-netlink resolve socket-default stroke updown
xauth-generic</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Virtual IP pools (size/online/offline):</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'><span style="mso-spacerun: yes;"> </span>10.0.0.0/8:
16777214/21757/0</span></div><div><font face="Times New Roman">
<br></font></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>The job queue load shows 2 jobs are queued for CRITICAL
priority. I also profiled the Charon daemon (IKE Responder) and found that,
most of the threads are getting blocked in pthread_cond_wait ().<span style="mso-spacerun: yes;"> </span>It means the thread currently has no work to
do and waits for a job. Any suggestion what limits this connection rate or any pointer
on how to debug this issue further?</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Regards,</span></div><div><font face="Times New Roman">
<br></font></div><div style="margin: 0in 0in 10pt;"><span style='line-height: 115%; font-family: "Verdana","sans-serif"; font-size: 10pt;'>Chinmaya</span></div><div><font face="Times New Roman">
<br></font></div></span><div><br></div><div class="yahoo_quoted" style="display: block;"> <br> <br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Friday, March 21, 2014 8:23 PM, Martin Willi <martin@strongswan.org> wrote:<br> </font> </div> <div class="y_msg_container"><br clear="none">> And the single receiver thread becomes bottleneck due to high<br clear="none">> connection rate/setup rate.<br clear="none"><br clear="none">The receiver job is rather trivial, only the IKE header is parsed and<br clear="none">some rate limiting is enforced for DoS protection. Any further<br clear="none">processing is delegated to the thread pool using that<br clear="none">process_message_job(). So I have my doubts that
this is the bottleneck<br clear="none">you are actually looking for.<br clear="none"><br clear="none">> Can it possible to create separate UDP sockets for each thread? The<br clear="none">> SO_REUSEPORT socket option allows multiple UDP sockets to be bound to<br clear="none">> the same port. With SO_REUSEPORT, multiple threads could use recvfrom<br clear="none">> () on its own socket to accept datagrams arriving on the port.<br clear="none"><br clear="none">Theoretically that is possible, but I'm not really sure if that helps to<br clear="none">fix the issues you are seeing.<br clear="none"><br clear="none">When running your tests, how does your job queue in "ipsec statusall"<br clear="none">look like? If you have many jobs queued that don't get processed,<br clear="none">something else prevents that charon scales properly on your platform.<div class="yqt2854752121" id="yqtfd27685"><br clear="none"><br clear="none">Regards</div><br
clear="none">Martin<div class="yqt2854752121" id="yqtfd91946"><br clear="none"><br clear="none"></div><br><br></div> </div> </div> </div> </div></body></html>