[strongSwan-dev] StrongSwan 5.6.3, Netlink performance issue as responder.

[Vinay G. Pullela]

Hi Tobias,

Thanks for the quick response.
Can you explain your below comment a little more
“You could try if the settings in the
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]
help, which configure the kernel to hash parts of the selectors of the
policies (how well that works depends on the negotiated traffic
selectors and the configured thresholds). Note that the size of the
policy hash table can't be configured, it's limited to 8 buckets (so if
you manage to hash policies you get eight lists instead of one, or nine
if you consider the overflow list for policies that aren't hashed due to
the thresholds).”

Are you talking about the below parameters
    kernel-netlink {
      spdh_thresh {
        ipv4 {
          lbits = 16
          rbits = 32
        }
        ipv6 {
          lbits = 128
          rbits = 128
        }
      }
    }

Is the Limit 8 for the policy table set in kernel?
What are the best values for the Hash to get to at least 100K tunnels at  200tunnels/sec.
And other kernel buffer parameters needs to be set from below list?

charon.plugins.kernel-netlink.buflen

min(PAGE_SIZE, 8192)

Buffer size for received Netlink messages.

charon.plugins.kernel-netlink.force_receive_buffer_size

no

If the maximum Netlink socket receive buffer in bytes set by receive_buffer_size exceeds the system-wide maximum from /proc/sys/net/core/rmem_max, this option can be used to override the limit. Enabling this option requires special priviliges (CAP_NET_ADMIN).

charon.plugins.kernel-netlink.fwmark

Firewall mark to set on the routing rule that directs traffic to our own routing table. The format is [!]mark[/mask], where the optional exclamation mark inverts the meaning (i.e. the rule only applies to packets that don't match the mark). A possible use case are host-to-host tunnels with kernel-libipsec<https://wiki.strongswan.org/projects/strongswan/wiki/Kernel-libipsec#Host-to-Host-Tunnels>. When set to !<mark> a more efficient lookup for source and next-hop addresses may also be used since 5.3.3<https://wiki.strongswan.org/versions/58>.

charon.plugins.kernel-netlink.mss

0

MSS to set on installed routes, 0 to disable.

charon.plugins.kernel-netlink.mtu

0

MTU to set on installed routes, 0 to disable.

charon.plugins.kernel-netlink.process_rules

no

Whether to process changes in routing rules to trigger roam events. This is currently only useful if the kernel based route lookup is used (i.e. if route installation is disabled or an inverted fwmark match is configured).

charon.plugins.kernel-netlink.receive_buffer_size

0

Maximum Netlink socket receive buffer in bytes. This value controls how many bytes of Netlink messages can be received on a Netlink socket. The default value is set by /proc/sys/net/core/rmem_default. The specified value cannot exceed the system-wide maximum from /proc/sys/net/core/rmem_max, unless force_receive_buffer_size is enabled.

charon.plugins.kernel-netlink.roam_events

yes

Whether to trigger roam events when interfaces, addresses or routes change.

charon.plugins.kernel-netlink.set_proto_port_transport_sa

no

Whether to set protocol and ports in the selector installed on transport mode IPsec SAs in the kernel. While doing so enforces policies for inbound traffic, it also prevents the use of a single IPsec SA by more than one traffic selector.

charon.plugins.kernel-netlink.spdh_thresh

Subsection to configure XFRM policy hashing thresholds for IPv4 and IPv6. The section defines hashing thresholds to configure in the kernel during daemon startup. Each address family takes a threshold for the local subnet of an IPsec policy (src in out-policies, dst in in- and forward-policies) and the remote subnet (dst in out-policies, src in in- and forward-policies).
If the subnet has more or equal net bits than the threshold, the first threshold bits are used to calculate a hash to lookup the policy.
Policy hashing thresholds are not supported before Linux 3.18 and might conflict with socket policies before Linux 4.8.

charon.plugins.kernel-netlink.spdh_thresh.ipv4.lbits

32

Local subnet XFRM policy hashing threshold for IPv4.

charon.plugins.kernel-netlink.spdh_thresh.ipv4.rbits

32

Remote subnet XFRM policy hashing threshold for IPv4.

charon.plugins.kernel-netlink.spdh_thresh.ipv6.lbits

128

Local subnet XFRM policy hashing threshold for IPv6.

charon.plugins.kernel-netlink.spdh_thresh.ipv6.rbits

128

Remote subnet XFRM policy hashing threshold for IPv6.



Thanks & Regards,
Vinay
From: Tobias Brunner <tobias at strongswan.org>
Sent: Tuesday, December 04, 2018 5:42 AM
To: Vinay G. Pullela <vpullela at parallelwireless.com>; dev at lists.strongswan.org
Subject: Re: [strongSwan-dev] StrongSwan 5.6.3, Netlink performance issue as responder.

Hi Vinay,

> What I mean by this is request and response for the kernel-netlink
> messages are taking more and more time as the number of established
> tunnels count, I have tried both with the parallel option enabled and
> disabled, see the observation for both below.

I'd say that's to be expected and probably related to how the kernel
manages policies. They are stored in a linear list, sorted by priority.
So every additional CHILD_SA takes longer to install because the
policies have to be inserted in the correct location in that list.

> 1. When parallel flag

I wouldn't expect this to change anything because the kernel has to lock
the policy lists and SA hash tables globally.

You could try if the settings in the
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]
help, which configure the kernel to hash parts of the selectors of the
policies (how well that works depends on the negotiated traffic
selectors and the configured thresholds). Note that the size of the
policy hash table can't be configured, it's limited to 8 buckets (so if
you manage to hash policies you get eight lists instead of one, or nine
if you consider the overflow list for policies that aren't hashed due to
the thresholds).

> Note: Route installation is disabled.

You mean via charon.install_routes? That's good, as installing one or
more routes for every client is another bottleneck with lots of clients.
And if e.g. virtual IPs are used, a single route can easily cover all
clients if it's even necessary.

Regards,
Tobias

[1] https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf<https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/dev/attachments/20181204/f1e19cd3/attachment-0001.html>