Martin, Guys,<br><br>Sorry to be a pain... but this behaviour is occurring every 6 hours or so now. This time there are no CLOSE_WAITS in the http server, suggesting that the initial cause is strongswan, ie when it falls over it affect other things.<br>
<br>I've added a cron job to restart ipsec every hour - I'll see if this solves things for now. I've flushed our iptables and xt_recent lists.<br><br>Are there any kernel patches / strongswan known issues? We're running a stock ubuntu-server kernel 2.6.32-24-server x86_64 (Ubuntu Lucid, 10.04) with strongSwan 4.3.2.<br>
<br>On restart, there are a whole swarm of plugin errors - probably due to the ubuntu packaging, but for completeness:-<br>Oct 26 06:00:12 api charon: 01[DMN] Starting IKEv2 charon daemon (strongSwan 4.3.2)<br>Oct 26 06:00:12 api charon: 01[NET] unable to create raw socket: Address family not supported by protocol<br>
Oct 26 06:00:12 api charon: 01[NET] could not open IPv6 receive socket, IPv6 disabled<br>Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'sha1' failed: /usr/lib/ipsec/plugins/libstrongswan-sha1.so: cannot open shared object file: No such file or directory<br>
Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'fips-prf' failed: /usr/lib/ipsec/plugins/libstrongswan-fips-prf.so: cannot open shared object file: No such file or directory<br>Oct 26 06:00:12 api charon: 01[KNL] listening on interfaces:<br>
Oct 26 06:00:12 api charon: 01[KNL] eth0<br>Oct 26 06:00:12 api charon: 01[KNL] eth1<br>Oct 26 06:00:12 api charon: 01[KNL] bond0<br>Oct 26 06:00:12 api charon: 01[KNL] 10.0.0.54<br>Oct 26 06:00:12 api charon: 01[KNL] xxx.20.214.10<br>
Oct 26 06:00:12 api charon: 01[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'<br>Oct 26 06:00:12 api charon: 01[LIB] loaded certificate file '/etc/ipsec.d/cacerts/ipsec-certificate-authority.certificate.pem'<br>
Oct 26 06:00:12 api charon: 01[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'<br>Oct 26 06:00:12 api charon: 01[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'<br>Oct 26 06:00:12 api charon: 01[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'<br>
Oct 26 06:00:12 api charon: 01[CFG] loading crls from '/etc/ipsec.d/crls'<br>Oct 26 06:00:12 api charon: 01[CFG] loading secrets from '/etc/ipsec.secrets'<br>Oct 26 06:00:12 api charon: 01[CFG] loaded private key file '/etc/ssh/ssh_host_rsa_key'<br>
Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'sql' failed: /usr/lib/ipsec/plugins/libstrongswan-sql.so: cannot open shared object file: No such file or directory<br>Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'attr' failed: /usr/lib/ipsec/plugins/libstrongswan-attr.so: cannot open shared object file: No such file or directory<br>
Oct 26 06:00:12 api charon: 01[CFG] no RADUIS secret defined<br>Oct 26 06:00:12 api charon: 01[CFG] RADIUS plugin initialization failed<br>Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'eapradius' failed: plugin_create() returned NULL<br>
Oct 26 06:00:12 api charon: 01[CFG] mediation database URI not defined, skipped<br>Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'medsrv' failed: plugin_create() returned NULL<br>Oct 26 06:00:12 api charon: 01[CFG] mediation client database URI not defined, skipped<br>
Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'medcli' failed: plugin_create() returned NULL<br>Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'nm' failed: /usr/lib/ipsec/plugins/libstrongswan-nm.so: cannot open shared object file: No such file or directory<br>
Oct 26 06:00:12 api charon: 01[LIB] loading plugin 'resolv-conf' failed: /usr/lib/ipsec/plugins/libstrongswan-resolv-conf.so: cannot open shared object file: No such file or directory<br>Oct 26 06:00:12 api charon: 01[DMN] loaded plugins: curl ldap random x509 pubkey openssl xcbc hmac agent gmp kernel-netlink stroke updown eapidentity eapmd5 eapgtc eapaka eapmschapv2 <br>
<br>BTW IPv6 is disabled in this server. I don't understand why strongswan should try listening on eth0 or eth1 - it should only be listening on bond0.<br><br>Raph<br><br clear="all">Raphael Cohn<br>Managing Director<br>
raphael.cohn@StormMQ.com<br>StormMQ Limited<br><br>UK Office: <br>Gateshead int'l Business Centre, Mulgrave Terrace, Gateshead, NE8 1AN, United Kingdom<br>Telephone: +44 845 3712 567<br><br>Registered office:<br>78 Broomfield Road, Chelmsford, Essex, CM1 1SS, United Kingdom<br>
StormMQ Limited is Registered in England and Wales under Company Number 07175657<br>StormMQ.com<br>
<br><br><div class="gmail_quote">On 25 October 2010 19:25, Raphael Cohn <span dir="ltr"><<a href="mailto:raphael.cohn@stormmq.com">raphael.cohn@stormmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Martin,<br><br>Thank you.<br><br>My config excerpts:-<br>config setup<br> crlcheckinterval=180<br> strictcrlpolicy=no<br> plutostart=no<br> charonstart=yes<br><br>ca ipsec-certificate-authority<br> cacert=ipsec-certificate-authority.certificate.pem<br>
auto=add<br><br>include /etc/ipsec.d/ipsec.mesh.conf<br><br>(ipsec.mesh.conf)<br>conn %default<br> ikelifetime=60m<br> keylife=20m<br> rekeymargin=3m<br> keyexchange=ikev2<br> # My IP address for default route (ie eth0)<br>
# We can not use %defaultroute (eg if using DHCP, or multiple routes because of an aliased interface)<br> left=10.0.0.54<br> leftcert=certificate.pem<br> leftsendcert=always<br> rightsendcert=always<br> auto=start<br>
<br><br>conn local-amqp<br> right=10.0.0.52<br> rightid="C=GB, ST=County Durham, L=Gateshead, O=StormMQ Limited, OU=<a href="http://amqp.stormmq.com" target="_blank">amqp.stormmq.com</a>, CN=<a href="http://amqp.stormmq.com" target="_blank">amqp.stormmq.com</a>"<br>
rightca="C=GB, ST=County Durham, L=Gateshead, O=StormMQ Limited, OU=<a href="http://stormmq.com" target="_blank">stormmq.com</a>, CN=<a href="http://stormmq.com" target="_blank">stormmq.com</a> IPSec Certificate Authority"<br>
<br>... others similar<br>
<br>We use bonded interfaces, bond0, with aliased ip addresses - in this case bond0 is the internal ip address 10.0.0.54.<br><br>We've identified a chain of events, in which many connections to our HTTP server end up in CLOSE_WAIT - this seems to happen because mid-connection an IP address is being blacklisted by iptables (using xt_recent module). At this point, connections in CLOSE_WAIT reach c. 500 or so and ipsec seems to then fall over... Blaclkisting occurs because a connection from a valid IP and http client is identified as 'Invalid' by iptables - as a solitary ACK packet initiating a connection. The source IP address is not one covered by ipsec routes.<br>
<br>It's not clear which causes what - iptables, HTTP server or ipsec. My suspicision is that CLOSE_WAITs occur because the http releasing a resource fails because of blacklisting - but why should ipsec's netlink code then fail subsequently?<br>
<br>Raph<br><br clear="all">Raphael Cohn<br>Managing Director<br>raphael.cohn@StormMQ.com<br>StormMQ Limited<br><br>UK Office: <br>Gateshead int'l Business Centre, Mulgrave Terrace, Gateshead, NE8 1AN, United Kingdom<br>
Telephone: +44 845 3712 567<br><br>Registered office:<br>78 Broomfield Road, Chelmsford, Essex, CM1 1SS, United Kingdom<br>StormMQ Limited is Registered in England and Wales under Company Number 07175657<br>StormMQ.com<div>
<div></div><div class="h5"><br>
<br><br><div class="gmail_quote">On 25 October 2010 08:39, Martin Willi <span dir="ltr"><<a href="mailto:martin@strongswan.org" target="_blank">martin@strongswan.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Raph,<br>
<div><br>
> creating rekey job for ESP CHILD_SA with SPI cbf0f0af and reqid {3720}<br>
</div><div>> creating rekey job for ESP CHILD_SA with SPI c3a35904 and reqid {3721}<br>
</div><div>> creating delete job for ESP CHILD_SA with SPI c2eb09ce and reqid {3720}<br>
</div><div>> creating delete job for ESP CHILD_SA with SPI cbf0f0af and reqid {3720}<br>
</div><div>> creating delete job for ESP CHILD_SA with SPI c3b5dc9e and reqid {3721}<br>
</div><div>> creating delete job for ESP CHILD_SA with SPI c3a35904 and reqid {3721}<br>
</div><div>> creating rekey job for ESP CHILD_SA with SPI c665f5aa and reqid {3722}<br>
</div><div>> creating rekey job for ESP CHILD_SA with SPI c1ccd29a and reqid {3718}<br>
</div>> ...<br>
<br>
The kernel triggers many rekey/delete events for the installed CHILD_SAs<br>
concurrently. What is your rekey configuration (lifetime/margin/fuzz)?<br>
<br>
Regards<br>
<font color="#888888">Martin<br>
<br>
</font></blockquote></div><br>
</div></div></blockquote></div><br>