[strongSwan] High charon CPU usage + stale SPIs in kernel

[Jiri Horky]

Hi list,

we have a rather specific setup with 102x102 host2host tunnels, each
host from left side conneting to all 102 hosts on the other side. We use
strongswan-5.2.0-2, we are on CentOS 6.5 with long term support kernel
v3.14.13. 

The problem we are seeing is that from time to time, a lot of SPIs are created:

[root at ams02-001 ~]# ip xfrm state | grep -c ^src
7857

Instead of expected ~ 102*2=204 SPIs. Also, we see a alot. Usually,

Some of the SPIs are in a stale state, on the affected nodes, we see this in charon.log:


Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI c663d950 failed: No such process (3)
Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI c5265bed failed: No such process (3)
Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI cea44e84 failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI c218d962 failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI c08f7af3 failed: No such process (3)
Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI ca3eb959 failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI ccd82ddf failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI c5e8e50d failed: No such process (3)
Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI c7c6f94f failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI cb94f74a failed: No such process (3)
Sep 18 10:15:57 13[KNL] <tch059|49961> querying SAD entry with SPI c2ebe4df failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI cc64998e failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI c68e87ce failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI cc044f83 failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI c38d0c03 failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI c433c8d0 failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI cd2d20c0 failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI c01a3729 failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI cb6871db failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI c5425a67 failed: No such process (3)
Sep 18 10:15:57 05[KNL] <tch059|49972> querying SAD entry with SPI c546ed9d failed: No such process (3)
Sep 18 10:15:57 07[KNL] <tch059|50015> querying SAD entry with SPI cab7ddd0 failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI c86040c9 failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI cc54935a failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI cb459320 failed: No such process (3)
Sep 18 10:15:57 12[KNL] <tch059|50021> querying SAD entry with SPI c471d8de failed: No such process (3)
Sep 18 10:15:57 06[KNL] <tch059|50061> querying SAD entry with SPI c4e7124f failed: No such process (3)
Sep 18 10:15:57 06[KNL] <tch059|50061> querying SAD entry with SPI c2c4b801 failed: No such process (3)

Some of the point to point assosiations are there multiple times:

ip xfrm state  | grep ^src | sort | uniq -c | sort -n

[root at ams02-001 ~]# ip xfrm state  | grep ^src | sort | uniq -c | sort -rn | head -n10
   3528 src 5.45.A.B dst 5.45.C.D
   3528 src 5.45.C.D dst 5.45.A.B
     88 src 5.45.A.B dst 5.45.E.F
     88 src 5.45.E.F dst 5.45.A.B

This might related to the kernel crashes we see which I described in thread "Occasional kernel crash at __xfrm_state_lookup".

Any help would be appreciated.

Jiri Horky