[strongSwan] system memory leak when soaking 1000 connections

[Simon Chan]

Greetings,

My versions:
- Debian 6.0.5
- kernel 3.0.23
- StrongSwan 4.4.1

In a large system test, I have one box serving 1000 road warriors. (Those 1000 road warriors are faked by another Linux box with 1000 leftid's,  1000 traffic selectors and 1000 ip aliases.)

After running for 2.5 days, free memory dropped by 69M. (Linux gurus may want to jump in here about Linux not include cache and buffers in free memory reporting. I am aware of that trap and is reading the really free memory.)

Entered the data in a spread sheet and plot a graph. The plot shows no sign of slowing down. So my VPN server may eventually bleed to death.
VM sizes of charon and all other processes were stable during the test period. Thus most likely the 69M was consumed by the kernel. Tracking kernel memory leak is a daunting task. Don't want to go there unless there are no other leads.

My line of thought, during the test period, only dead-peer-detection and key renewal were running the whole time. DPD should not be the cause since charon memory footprint was stable. I wonder if IKE channel renewal and key renewal may leave entries pilling up in Security Policy db and Security Association db?
I ran a test with ikelifetime and keylife set to 5hr with rekeymargin=10m. The 1000 connections are established 2 seconds apart. Renewing 1000 tunnels take 33.3 minutes. Watching free memory closely at renewal time, I saw a steep drop during the 33m interval. Although after renwal was over, system memory was still decreasing for the next 4.25 hrs but at a much slower pace.

I tried reading the debug log on a single connection key renewal. As far as I can tell charon makes all necessary calls to cleanup SPD and SAD.
I am lost how to explain the corelation between key renewal and memory drop. Anyone have any ideas how to debug this problem?

BTW just finished running the test on 2.6.37 kernel and it showed the same memory lost pattern.

Regards,
Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20130123/de018ab1/attachment.html>