<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12pt"><div><span></span></div><div class="yahoo_quoted" style="display: block;"> <br> <br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Wednesday, March 26, 2014 5:24 PM, Chinmaya Dwibedy <ckdwibedy@yahoo.com> wrote:<br> </font> </div> <div class="y_msg_container"><div id="yiv3576163924"><div><div style="color: rgb(0, 0, 0); font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><div><span><br clear="none"></span></div><div><br
clear="none"></div><span></span><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Hi, </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Thanks again for your valuable suggestion. I request your
goodness to go thru the below stated and guide me . </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">We are using two Multi-Core MIPS64 Processors with 16 cnMIPS64
v2 cores (one acts as an IKE initiator and another as an IKE responder). We are
running strongswan in both systems. Both the systems have 1Gbps Ethernet cards,
which are connected to 1 Gbps L2 switch. The Wind River Linux (kernel with SMP
support) runs on all the 16 cores.</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">I have restricted one instance of starter/charon to run on
first core but the group of threads (created and managed by the strongswan
based upon configuration setting in strongswan.conf file) in order to process a
large number of tasks, will be scheduled and distributed among 16 cores. Being
multi-threaded charon should scale very well to multiple cores, taking full
advantage of multi-core systems.</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">In peak load (250k IPsec sessions), I run #perf top , which
provides a real-time aggregate of functions where most of the time is spent
across all cpu's and processes. Here goes the call stacks at both ends.</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">IKE Initiator</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;">-------------------------------------------------------------------------------</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>PerfTop:<span> </span>846707 irqs/sec<span> </span>kernel:88.3% [1000Hz cpu-clock-msecs],<span> </span>(all, 16 CPUs)</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">-------------------------------------------------------------------------------</span></div><div><font face="Times New Roman">
</font><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>samples<span> </span>pcnt function<span> </span>DSO</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>_______
_____ ________________________ ___________________________</span></div><div><font face="Times New Roman">
<br></font></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>9323523.00
89.7% r4k_wait<span>
</span>[kernel.kallsyms]</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>173541.00<span> </span>1.7%
dso__find_symbol<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>101710.00<span> </span>1.0%
pthread_mutex_lock<span> </span>/lib64/libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>85056.00<span> </span>0.8%
event__preprocess_sample /usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>48367.00<span> </span>0.5% vfprintf<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>44368.00<span> </span>0.4%
pthread_rwlock_rdlock<span>
</span>/lib64/libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>41310.00 <span> </span>0.4% __libc_malloc<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>40884.00<span> </span>0.4%
__pthread_rwlock_unlock<span>
</span>/lib64/libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>34031.00<span> </span>0.3% maps__find<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>29452.00<span> </span>0.3% cfree<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>26343.00<span> </span>0.3% dump_printf<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>20629.00<span> </span>0.2%
perf_session__findnew<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>18454.00<span> </span>0.2%
map__find_symbol<span> </span><span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>15415.00<span> </span>0.1%
finish_task_switch<span>
</span>[kernel.kallsyms]</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>13397.00<span> </span>0.1%
_IO_default_xsputn<span>
</span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
</font><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">IKE Responder</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;">PerfTop:<span> </span>869276
irqs/sec<span> </span>kernel:88.8% [1000Hz
cpu-clock-msecs],<span> </span>(all, 16 CPUs)</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">-------------------------------------------------------------------------------</span></div><div><font face="Times New Roman">
</font><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>samples<span> </span>pcnt function<span> </span>DSO</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>_______
_____ ___________________________ _____________________</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>10201775.00
89.5% r4k_wait<span>
</span>[kernel.kallsyms]</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>180167.00<span> </span>1.6%
dso__find_symbol<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>109755.00<span> </span>1.0%
pthread_mutex_lock<span>
</span>libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>91641.00<span> </span>0.8%
event__preprocess_sample<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span><span> </span>86032.00<span> </span>0.8% pthread_rwlock_rdlock<span> </span>libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>79349.00<span> </span>0.7%
__pthread_rwlock_unlock<span>
</span>libpthread-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>41658.00<span> </span>0.4% cfree<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>40051.00<span> </span>0.4% __libc_malloc<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>37084.00<span> </span>0.3% maps__find<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>36666.00<span> </span>0.3% vfprintf<span> </span>/lib64/libc-2.11.1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>28505.00<span> </span>0.2% dump_printf<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>27656.00<span> </span>0.2%
finish_task_switch<span>
</span>[kernel.kallsyms]</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>24894.00<span> </span>0.2% SHA1Transform<span> </span>libstrongswan-sha1.so</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>20652.00<span> </span>0.2%
perf_session__findnew<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>20321.00<span> </span>0.2%
map__find_symbol<span> </span>/usr/bin/perf</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>15616.00<span> </span>0.1%
_raw_spin_unlock_irqrestore [kernel.kallsyms]</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">The overall CPU utilization at both ends was below 10%. I
profiled (using perf tool) the some threads at both the ends to figure out the hotspots.The
profiled result shows that, it spends most of the time in
__pthread_rwlock_rdlock(). The #mpstat –P ALL command (at both the ends) shows
that, the core 0 is fully busy, while other 15 cores are simply idles. </span><span style="line-height: 115%;">Since all of threads are getting executed on the same CPU (Core 0) , Charon
is not scaling well. </span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Here goes the strongswan configuration at both the ends</span></div><div><font face="Times New Roman">
<br></font></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">IKE Initiator </span></div><div><font face="Times New Roman">
</font><font face="Times New Roman">
</font><span style="line-height: 115%;"><span> </span>threads = 32</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>replay_window =
32</span></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"></span><span style="line-height: 115%;"><span> </span>dos_protection =
no</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>block_threshold=1000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>cookie_threshold=1000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>init_limit_half_open=25000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt; text-indent: 0.5in;"><span style="line-height: 115%;">init_limit_job_load=25000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>retransmit_timeout=30</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>retransmit_tries=30</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>install_virtual_ip=no</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>install_routes=no</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>close_ike_on_child_failure=yes</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>ikesa_table_size
= 73728</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>ikesa_table_segments = 256</span></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"></span><span style="line-height: 115%;"><span> </span>reuse_ikesa = no</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>load-tester {</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>enable = yes</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>initiators = 5</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>iterations = 50000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>delay
= 10</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>responder = 30.30.30.21</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>proposal = aes128-sha1-modp768</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>initiator_auth = psk</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>responder_auth = psk</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>request_virtual_ip = yes</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>initiator_tsr=40.0.0.0/8</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>ike_rekey = 0</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>child_rekey = 0</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span><span> </span>delete_after_established = no</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>shutdown_when_complete = no</span></div><div><font face="Times New Roman">
</font><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">libstrongswan {</span><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
</font><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>dh_exponent_ansi_x9_42 = no</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">}</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">IKE Responder</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span> </span>threads = 32</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>replay_window =
32</span></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"></span><span style="line-height: 115%;"><span> </span>dos_protection =
no</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>block_threshold=1000</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>cookie_threshold=1000</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span>
</span>init_limit_half_open=25000</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span>
</span>init_limit_job_load=25000</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span>
</span>half_open_timeout=1000</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span>
</span>close_ike_on_child_failure=yes</span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;"><span> </span>ikesa_table_size
= 73728</span><font face="Times New Roman"><br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span>
</span>ikesa_table_segments = 256</span></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"></span><span style="line-height: 115%;"><span> </span>reuse_ikesa = no</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">libstrongswan {</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>dh_exponent_ansi_x9_42
= no</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>processor {</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>priority_threads
{</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>high = 1</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>medium = 10</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>}</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>}</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">}</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Also I compiled with --enable-lock-profiler and run with --nofork
with 250k IPsec session. But during daemon shutdown, it does not print the
cumulative time waited in each lock to stderr. </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"> </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Regards,</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Chinmaya </span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div><br clear="none"></div><div class="yiv3576163924yahoo_quoted" style="display: block;"> <br clear="none"> <br clear="none"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div class="yiv3576163924yqt1507094357" id="yiv3576163924yqt11192"><div dir="ltr"> <font face="Arial" size="2"> On Monday, March 24, 2014 3:11 PM, Chinmaya Dwibedy <ckdwibedy@yahoo.com> wrote:<br clear="none"> </font> </div> <div class="yiv3576163924y_msg_container"><div id="yiv3576163924"><div><div style="color: rgb(0, 0, 0); font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><div><span><br clear="none"></span></div><div><br
clear="none"></div><span></span><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Hi Martin,</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Thanks a lot Martin for your prompt response and valuable
suggestion as well. </span></div><div><font face="Times New Roman">
</font><span style="line-height: 115%;">I have configured the number of threads to 32 at both
the ends (IKE responder and IKE Initiator). At IKE Initiator end, if I increase
the sender threads (i.e., initiators in load-tester section) from 5 to 10
(i.e., to put load on all cores), I find the followings at IKE Responder end.</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><ol style="list-style-type: decimal; direction: ltr;"><li style="color: rgb(0, 0, 0); font-size: 10pt; font-style: normal; font-weight: normal;"><div style="color: rgb(0, 0, 0); font-size: 11pt; font-style: normal; font-weight: normal; margin-top: 0in; margin-bottom: 0pt;"><span style="line-height: 115%;">There are huge packets loss (i.e., packet
receive errors in the Udp section) in #netstat –su.</span></div></li><li style="color: rgb(0, 0, 0);"><div style="color: rgb(0, 0, 0);"><span style="line-height: 115%;">The Recv-Q column of the isakmp connection
is high and doesn't drop to zero in<span>
</span>#netstat –ua.</span></div></li></ol><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">From the above statistics, it just means that Receiver
thread ( of Charon daemon at IKE Responder end) is not reading the socket fast
enough. The average arrival rate regularly causes a backlog in the receive
queue. The maximum number of queued received data depends on
/proc/sys/net/ipv4/udp_mem and /proc/sys/net/ipv4/udp_rmem_min. Thus I increased
these values to 8MB before the start of test but still faced the same issue.
Note that, if I keep the initiators and iterations to 5 and 50000 respectively,
then there was no loss at all and I am getting the setup rate to be
220-225<span> </span></span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Here goes the output of “ipsec statusall" at IKE
responder end during test?</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Status of IKE charon daemon (strongSwan 5.0.4, Linux
2.6.34.10-grsec-BenuOcteon,</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>mips64):</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>uptime: 2
minutes, since Jan 01 00:11:15 1970</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>malloc: sbrk
500166656, mmap 5005312, used 500018224, free 148432</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>worker threads:
24 of 32 idle, 6/2/0/0 working, job queue: 0/2/0/0, scheduled:</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>37639</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>loaded plugins:
charon aes des sha1 sha2 md5 random nonce x509 revocation cons</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">traints pubkey pkcs1 pkcs8 pgp dnskey pem fips-prf gmp
xcbc cmac hmac attr kerne</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">l-netlink resolve socket-default stroke updown
xauth-generic</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Virtual IP pools (size/online/offline):</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;"><span> </span>10.0.0.0/8:
16777214/21757/0</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">The job queue load shows 2 jobs are queued for CRITICAL
priority. I also profiled the Charon daemon (IKE Responder) and found that,
most of the threads are getting blocked in pthread_cond_wait ().<span> </span>It means the thread currently has no work to
do and waits for a job. Any suggestion what limits this connection rate or any pointer
on how to debug this issue further?</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Regards,</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div style="margin: 0in 0in 10pt;"><span style="line-height: 115%;">Chinmaya</span></div><div><font face="Times New Roman">
<br clear="none"></font></div><div><br clear="none"></div><div class="yiv3576163924yqt6596794958" id="yiv3576163924yqt96399"><div class="yiv3576163924yahoo_quoted" style="display: block;"> <br clear="none"> <br clear="none"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Friday, March 21, 2014 8:23 PM, Martin Willi <martin@strongswan.org> wrote:<br clear="none"> </font> </div> <div class="yiv3576163924y_msg_container"><br clear="none">> And the single receiver thread becomes bottleneck due to high<br clear="none">> connection rate/setup rate.<br clear="none"><br clear="none">The receiver job is rather trivial, only the IKE header is parsed and<br clear="none">some rate limiting is enforced for DoS
protection. Any further<br clear="none">processing is delegated to the thread pool using that<br clear="none">process_message_job(). So I have my doubts that
this is the bottleneck<br clear="none">you are actually looking for.<br clear="none"><br clear="none">> Can it possible to create separate UDP sockets for each thread? The<br clear="none">> SO_REUSEPORT socket option allows multiple UDP sockets to be bound to<br clear="none">> the same port. With SO_REUSEPORT, multiple threads could use recvfrom<br clear="none">> () on its own socket to accept datagrams arriving on the port.<br clear="none"><br clear="none">Theoretically that is possible, but I'm not really sure if that helps to<br clear="none">fix the issues you are seeing.<br clear="none"><br clear="none">When running your tests, how does your job queue in "ipsec statusall"<br clear="none">look like? If you have many jobs queued that don't get processed,<br clear="none">something else prevents that charon scales properly on your platform.<div class="yiv3576163924yqt2854752121" id="yiv3576163924yqtfd27685"><br clear="none"><br
clear="none">Regards</div><br clear="none">Martin<div class="yiv3576163924yqt2854752121" id="yiv3576163924yqtfd91946"><br clear="none"><br clear="none"></div><br clear="none"><br clear="none"></div> </div> </div> </div></div> </div></div></div><br clear="none"><div class="yiv3576163924yqt6596794958" id="yiv3576163924yqt69338">_______________________________________________<br clear="none">Users mailing list<br clear="none"><a href="mailto:Users@lists.strongswan.org" target="_blank" rel="nofollow" shape="rect" ymailto="mailto:Users@lists.strongswan.org">Users@lists.strongswan.org</a><br clear="none"><a href="https://lists.strongswan.org/mailman/listinfo/users" target="_blank" rel="nofollow" shape="rect">https://lists.strongswan.org/mailman/listinfo/users</a></div><br clear="none"><br clear="none"></div></div> </div> </div> </div> </div></div></div><br><br></div> </div> </div> </div> </div></body></html>