<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Tobias,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for the quick response.<o:p></o:p></p>
<p class="MsoNormal">Can you explain your below comment a little more<o:p></o:p></p>
<p class="MsoNormal">“You could try if the settings in the<br>
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]<br>
help, which configure the kernel to hash parts of the selectors of the<br>
policies (how well that works depends on the negotiated traffic<br>
selectors and the configured thresholds). Note that the size of the<br>
policy hash table can't be configured, it's limited to 8 buckets (so if<br>
you manage to hash policies you get eight lists instead of one, or nine<br>
if you consider the overflow list for policies that aren't hashed due to<br>
the thresholds).”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Are you talking about the below parameters<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> kernel-netlink {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> spdh_thresh {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ipv4 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> lbits = 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> rbits = 32<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ipv6 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> lbits = 128<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> rbits = 128<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">Is the Limit 8 for the policy table set in kernel?<o:p></o:p></p>
<p class="MsoNormal">What are the best values for the Hash to get to at least 100K tunnels at 200tunnels/sec.<o:p></o:p></p>
<p class="MsoNormal">And other kernel buffer parameters needs to be set from below list?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellspacing="3" cellpadding="0" style="border:solid windowtext 1.0pt">
<tbody>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.buflen <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">min(PAGE_SIZE, 8192)<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Buffer size for received Netlink messages.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.force_receive_buffer_size<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">no <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">If the maximum Netlink socket receive buffer in bytes set by
<i>receive_buffer_size</i> exceeds the system-wide maximum from <span style="font-size:10.0pt;font-family:"Courier New"">
/proc/sys/net/core/rmem_max</span>, this option can be used to override the limit. Enabling this option requires special priviliges (CAP_NET_ADMIN).<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.fwmark <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt"></td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Firewall mark to set on the routing rule that directs traffic to our own routing table. The format is [!]mark[/mask], where the optional exclamation mark inverts the meaning (i.e. the rule only applies to packets that don't match the mark).
A possible use case are <a href="https://wiki.strongswan.org/projects/strongswan/wiki/Kernel-libipsec#Host-to-Host-Tunnels">
host-to-host tunnels with kernel-libipsec</a>. When set to <i>!<mark></i> a more efficient lookup for source and next-hop addresses may also be used since
<a href="https://wiki.strongswan.org/versions/58">5.3.3</a>.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.mss <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">0 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">MSS to set on installed routes, 0 to disable.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.mtu <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">0 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">MTU to set on installed routes, 0 to disable.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.process_rules <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">no <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Whether to process changes in routing rules to trigger roam events. This is currently only useful if the kernel based route lookup is used (i.e. if route installation is disabled or an inverted fwmark match is configured).<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.receive_buffer_size <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">0 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Maximum Netlink socket receive buffer in bytes. This value controls how many bytes of Netlink messages can be received on a Netlink socket. The default value is set by
<span style="font-size:10.0pt;font-family:"Courier New"">/proc/sys/net/core/rmem_default</span>. The specified value cannot exceed the system-wide maximum from
<span style="font-size:10.0pt;font-family:"Courier New"">/proc/sys/net/core/rmem_max</span>, unless
<i>force_receive_buffer_size</i> is enabled.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.roam_events <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">yes <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Whether to trigger roam events when interfaces, addresses or routes change.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.set_proto_port_transport_sa<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">no <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Whether to set protocol and ports in the selector installed on transport mode IPsec SAs in the kernel. While doing so enforces policies for inbound traffic, it also prevents the use of a single IPsec SA by more than one traffic selector.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.spdh_thresh <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt"></td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Subsection to configure XFRM policy hashing thresholds for IPv4 and IPv6. The section defines hashing thresholds to configure in the kernel during daemon startup. Each address family takes a threshold for the local subnet of an IPsec policy
(src in out-policies, dst in in- and forward-policies) and the remote subnet (dst in out-policies, src in in- and forward-policies).<br>
If the subnet has more or equal net bits than the threshold, the first threshold bits are used to calculate a hash to lookup the policy.<br>
Policy hashing thresholds are not supported before Linux 3.18 and might conflict with socket policies before Linux 4.8.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.spdh_thresh.ipv4.lbits<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">32 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Local subnet XFRM policy hashing threshold for IPv4.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.spdh_thresh.ipv4.rbits<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">32 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Remote subnet XFRM policy hashing threshold for IPv4.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.spdh_thresh.ipv6.lbits<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">128 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Local subnet XFRM policy hashing threshold for IPv6.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">charon.plugins.kernel-netlink.spdh_thresh.ipv6.rbits<o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">128 <o:p></o:p></p>
</td>
<td style="border:solid windowtext 1.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Remote subnet XFRM policy hashing threshold for IPv6.<o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks & Regards,<o:p></o:p></p>
<p class="MsoNormal">Vinay<o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Tobias Brunner <tobias@strongswan.org> <br>
<b>Sent:</b> Tuesday, December 04, 2018 5:42 AM<br>
<b>To:</b> Vinay G. Pullela <vpullela@parallelwireless.com>; dev@lists.strongswan.org<br>
<b>Subject:</b> Re: [strongSwan-dev] StrongSwan 5.6.3, Netlink performance issue as responder.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Vinay,<br>
<br>
> What I mean by this is request and response for the kernel-netlink<br>
> messages are taking more and more time as the number of established<br>
> tunnels count, I have tried both with the parallel option enabled and<br>
> disabled, see the observation for both below.<br>
<br>
I'd say that's to be expected and probably related to how the kernel<br>
manages policies. They are stored in a linear list, sorted by priority.<br>
So every additional CHILD_SA takes longer to install because the<br>
policies have to be inserted in the correct location in that list.<br>
<br>
> 1. When parallel flag<br>
<br>
I wouldn't expect this to change anything because the kernel has to lock<br>
the policy lists and SA hash tables globally.<br>
<br>
You could try if the settings in the<br>
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]<br>
help, which configure the kernel to hash parts of the selectors of the<br>
policies (how well that works depends on the negotiated traffic<br>
selectors and the configured thresholds). Note that the size of the<br>
policy hash table can't be configured, it's limited to 8 buckets (so if<br>
you manage to hash policies you get eight lists instead of one, or nine<br>
if you consider the overflow list for policies that aren't hashed due to<br>
the thresholds).<br>
<br>
> Note: Route installation is disabled.<br>
<br>
You mean via charon.install_routes? That's good, as installing one or<br>
more routes for every client is another bottleneck with lots of clients.<br>
And if e.g. virtual IPs are used, a single route can easily cover all<br>
clients if it's even necessary.<br>
<br>
Regards,<br>
Tobias<br>
<br>
[1] <a href="https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf">
https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf</a><o:p></o:p></p>
</div>
</body>
</html>