[strongSwan] Using multiple UDP sockets with SO_REUSEPORT option to increase high connection rate
Chinmaya Dwibedy
ckdwibedy at yahoo.com
Wed Mar 26 13:01:29 CET 2014
On Wednesday, March 26, 2014 5:24 PM, Chinmaya Dwibedy <ckdwibedy at yahoo.com> wrote:
Hi,
Thanks again for your valuable suggestion. I request your
goodness to go thru the below stated and guide me .
We are using two Multi-Core MIPS64 Processors with 16 cnMIPS64
v2 cores (one acts as an IKE initiator and another as an IKE responder). We are
running strongswan in both systems. Both the systems have 1Gbps Ethernet cards,
which are connected to 1 Gbps L2 switch. The Wind River Linux (kernel with SMP
support) runs on all the 16 cores.
I have restricted one instance of starter/charon to run on
first core but the group of threads (created and managed by the strongswan
based upon configuration setting in strongswan.conf file) in order to process a
large number of tasks, will be scheduled and distributed among 16 cores. Being
multi-threaded charon should scale very well to multiple cores, taking full
advantage of multi-core systems.
In peak load (250k IPsec sessions), I run #perf top , which
provides a real-time aggregate of functions where most of the time is spent
across all cpu's and processes. Here goes the call stacks at both ends.
IKE Initiator
-------------------------------------------------------------------------------
PerfTop: 846707 irqs/sec kernel:88.3% [1000Hz cpu-clock-msecs], (all, 16 CPUs)
-------------------------------------------------------------------------------
samples pcnt function DSO
_______
_____ ________________________ ___________________________
9323523.00
89.7% r4k_wait [kernel.kallsyms]
173541.00 1.7%
dso__find_symbol /usr/bin/perf
101710.00 1.0%
pthread_mutex_lock /lib64/libpthread-2.11.1.so
85056.00 0.8%
event__preprocess_sample /usr/bin/perf
48367.00 0.5% vfprintf /lib64/libc-2.11.1.so
44368.00 0.4%
pthread_rwlock_rdlock /lib64/libpthread-2.11.1.so
41310.00 0.4% __libc_malloc /lib64/libc-2.11.1.so
40884.00 0.4%
__pthread_rwlock_unlock /lib64/libpthread-2.11.1.so
34031.00 0.3% maps__find /usr/bin/perf
29452.00 0.3% cfree /lib64/libc-2.11.1.so
26343.00 0.3% dump_printf /usr/bin/perf
20629.00 0.2%
perf_session__findnew /usr/bin/perf
18454.00 0.2%
map__find_symbol /usr/bin/perf
15415.00 0.1%
finish_task_switch [kernel.kallsyms]
13397.00 0.1%
_IO_default_xsputn /lib64/libc-2.11.1.so
IKE Responder
PerfTop: 869276
irqs/sec kernel:88.8% [1000Hz
cpu-clock-msecs], (all, 16 CPUs)
-------------------------------------------------------------------------------
samples pcnt function DSO
_______
_____ ___________________________ _____________________
10201775.00
89.5% r4k_wait [kernel.kallsyms]
180167.00 1.6%
dso__find_symbol /usr/bin/perf
109755.00 1.0%
pthread_mutex_lock libpthread-2.11.1.so
91641.00 0.8%
event__preprocess_sample /usr/bin/perf
86032.00 0.8% pthread_rwlock_rdlock libpthread-2.11.1.so
79349.00 0.7%
__pthread_rwlock_unlock libpthread-2.11.1.so
41658.00 0.4% cfree /lib64/libc-2.11.1.so
40051.00 0.4% __libc_malloc /lib64/libc-2.11.1.so
37084.00 0.3% maps__find /usr/bin/perf
36666.00 0.3% vfprintf /lib64/libc-2.11.1.so
28505.00 0.2% dump_printf /usr/bin/perf
27656.00 0.2%
finish_task_switch [kernel.kallsyms]
24894.00 0.2% SHA1Transform libstrongswan-sha1.so
20652.00 0.2%
perf_session__findnew /usr/bin/perf
20321.00 0.2%
map__find_symbol /usr/bin/perf
15616.00 0.1%
_raw_spin_unlock_irqrestore [kernel.kallsyms]
The overall CPU utilization at both ends was below 10%. I
profiled (using perf tool) the some threads at both the ends to figure out the hotspots.The
profiled result shows that, it spends most of the time in
__pthread_rwlock_rdlock(). The #mpstat –P ALL command (at both the ends) shows
that, the core 0 is fully busy, while other 15 cores are simply idles. Since all of threads are getting executed on the same CPU (Core 0) , Charon
is not scaling well.
Here goes the strongswan configuration at both the ends
IKE Initiator
threads = 32
replay_window =
32
dos_protection =
no
block_threshold=1000
cookie_threshold=1000
init_limit_half_open=25000
init_limit_job_load=25000
retransmit_timeout=30
retransmit_tries=30
install_virtual_ip=no
install_routes=no
close_ike_on_child_failure=yes
ikesa_table_size
= 73728
ikesa_table_segments = 256
reuse_ikesa = no
load-tester {
enable = yes
initiators = 5
iterations = 50000
delay
= 10
responder = 30.30.30.21
proposal = aes128-sha1-modp768
initiator_auth = psk
responder_auth = psk
request_virtual_ip = yes
initiator_tsr=40.0.0.0/8
ike_rekey = 0
child_rekey = 0
delete_after_established = no
shutdown_when_complete = no
libstrongswan {
dh_exponent_ansi_x9_42 = no
}
IKE Responder
threads = 32
replay_window =
32
dos_protection =
no
block_threshold=1000
cookie_threshold=1000
init_limit_half_open=25000
init_limit_job_load=25000
half_open_timeout=1000
close_ike_on_child_failure=yes
ikesa_table_size
= 73728
ikesa_table_segments = 256
reuse_ikesa = no
libstrongswan {
dh_exponent_ansi_x9_42
= no
processor {
priority_threads
{
high = 1
medium = 10
}
}
}
Also I compiled with --enable-lock-profiler and run with --nofork
with 250k IPsec session. But during daemon shutdown, it does not print the
cumulative time waited in each lock to stderr.
Regards,
Chinmaya
On Monday, March 24, 2014 3:11 PM, Chinmaya Dwibedy <ckdwibedy at yahoo.com> wrote:
Hi Martin,
Thanks a lot Martin for your prompt response and valuable
suggestion as well.
I have configured the number of threads to 32 at both
the ends (IKE responder and IKE Initiator). At IKE Initiator end, if I increase
the sender threads (i.e., initiators in load-tester section) from 5 to 10
(i.e., to put load on all cores), I find the followings at IKE Responder end.
1. There are huge packets loss (i.e., packet
receive errors in the Udp section) in #netstat –su.
2. The Recv-Q column of the isakmp connection
is high and doesn't drop to zero in #netstat –ua.
From the above statistics, it just means that Receiver
thread ( of Charon daemon at IKE Responder end) is not reading the socket fast
enough. The average arrival rate regularly causes a backlog in the receive
queue. The maximum number of queued received data depends on
/proc/sys/net/ipv4/udp_mem and /proc/sys/net/ipv4/udp_rmem_min. Thus I increased
these values to 8MB before the start of test but still faced the same issue.
Note that, if I keep the initiators and iterations to 5 and 50000 respectively,
then there was no loss at all and I am getting the setup rate to be
220-225
Here goes the output of “ipsec statusall" at IKE
responder end during test?
Status of IKE charon daemon (strongSwan 5.0.4, Linux
2.6.34.10-grsec-BenuOcteon,
mips64):
uptime: 2
minutes, since Jan 01 00:11:15 1970
malloc: sbrk
500166656, mmap 5005312, used 500018224, free 148432
worker threads:
24 of 32 idle, 6/2/0/0 working, job queue: 0/2/0/0, scheduled:
37639
loaded plugins:
charon aes des sha1 sha2 md5 random nonce x509 revocation cons
traints pubkey pkcs1 pkcs8 pgp dnskey pem fips-prf gmp
xcbc cmac hmac attr kerne
l-netlink resolve socket-default stroke updown
xauth-generic
Virtual IP pools (size/online/offline):
10.0.0.0/8:
16777214/21757/0
The job queue load shows 2 jobs are queued for CRITICAL
priority. I also profiled the Charon daemon (IKE Responder) and found that,
most of the threads are getting blocked in pthread_cond_wait (). It means the thread currently has no work to
do and waits for a job. Any suggestion what limits this connection rate or any pointer
on how to debug this issue further?
Regards,
Chinmaya
On Friday, March 21, 2014 8:23 PM, Martin Willi <martin at strongswan.org> wrote:
> And the single receiver thread becomes bottleneck due to high
> connection rate/setup rate.
The receiver job is rather trivial, only the IKE header is parsed and
some rate limiting is enforced for DoS
protection. Any further
processing is delegated to the thread pool using that
process_message_job(). So I have my doubts that
this is the bottleneck
you are actually looking for.
> Can it possible to create separate UDP sockets for each thread? The
> SO_REUSEPORT socket option allows multiple UDP sockets to be bound to
> the same port. With SO_REUSEPORT, multiple threads could use recvfrom
> () on its own socket to accept datagrams arriving on the port.
Theoretically that is possible, but I'm not really sure if that helps to
fix the issues you are seeing.
When running your tests, how does your job queue in "ipsec statusall"
look like? If you have many jobs queued that don't get processed,
something else prevents that charon scales properly on your platform.
Regards
Martin
_______________________________________________
Users mailing list
Users at lists.strongswan.org
https://lists.strongswan.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20140326/d2632b30/attachment-0001.html>
More information about the Users
mailing list