[strongSwan] Occasional kernel crash at __xfrm_state_lookup

Jiri Horky jiri.horky at gmail.com
Thu Sep 18 10:09:04 CEST 2014


Hi list,

we have a rather specific setup with 102x102 host2host tunnels, each
host from left side conneting to all 102 hosts to the other side. We use
strongswan-5.2.0-2, we are on CentOS 6.5 with long term support kernel
v3.14.13. We switched to this kernel after experiencing similar crashes
on stock 2.6.32-431.11.2.el6.x86_64 kernel.

The configuration looks like follows:

[root at ams05-004 127.0.0.1-2014-09-17-21:36:11]# cat
/etc/strongswan/ipsec.conf
config setup
        uniqueids=no

conn %default
        ikelifetime=180m
        keylife=90m
        rekeymargin=5m
        keyingtries=%forever
        keyexchange=ikev2
        mobike=no
        reauth=no
        dpdaction=restart
        closeaction=restart

include /etc/strongswan/ipsec.d/*.conf

One of the connection file looks like this:
conn tch017
        left=5.45.X.Y
        right=5.45.Z.W
        authby=secret
        type=transport
        auto=route

On each of the hosts, we run periodical ping to all hosts on the other
side of the tunnels so we keep the tunnels up.

We experience relatively frequent kernel crash (~2 a day out of 102
nodes). I enclose a stack trace:

[17243.492008] general protection fault: 0000 [#1] SMP
[17243.492038] Modules linked in: nfsv3 nfs_acl ipmi_si vfat fat
usb_storage mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase
ipmi_devintf dell_rbu authenc xfrm4_mode_transport cbc xfrm4_tunnel
tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key rpcsec_gss_krb5 auth_rpcgss
nfsv4 nfs fscache lockd sunrpc cls_u32 sch_prio 8021q garp stp llc
bonding xt_comment iptable_filter iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_DSCP iptable_mangle xt_CT xt_set
iptable_raw ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ip_set_hash_net
ip_set nfnetlink fuse dm_mirror dm_region_hash dm_log dm_mod gpio_ich
iTCO_wdt iTCO_vendor_support dcdbas microcode pcspkr sb_edac edac_core
mgag200 ttm drm_kms_helper sysimgblt sysfillrect syscopyarea
[17243.492370]  ipmi_msghandler acpi_pad lpc_ich shpchp joydev sg tg3
ptp pps_core wmi acpi_power_meter hwmon ext4 jbd2 mbcache sd_mod
crc_t10dif crct10dif_common megaraid_sas [last unloaded: ipmi_si]
[17243.492449] CPU: 12 PID: 16085 Comm: charon Not tainted
3.14.13.ffdebug2 #8
[17243.492473] Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS
2.2.2 01/16/2014
[17243.492499] task: ffff883fe0948e30 ti: ffff883fdd728000 task.ti:
ffff883fdd728000
[17243.492525] RIP: 0010:[<ffffffff81601b18>]  [<ffffffff81601b18>]
__xfrm_state_lookup+0x78/0x100
[17243.492559] RSP: 0018:ffff883fdd7299f0  EFLAGS: 00010202
[17243.492577] RAX: 0060000000000804 RBX: 000000000000012e RCX:
0000000017cd52c6
[17243.492605] RDX: ffff880063f05810 RSI: 0000000000000000 RDI:
ffff880a2552c028
[17243.492629] RBP: ffff883fdd7299f8 R08: 0000000000000032 R09:
0000000000000002
[17243.492654] R10: ffff880a2552c000 R11: 000000000007ffff R12:
ffffffff81d0ef40
[17243.492678] R13: 0000000000000000 R14: ffff880063f05810 R15:
0000000017cd52c6
[17243.492702] FS:  00007f4e61c89700(0000) GS:ffff88207fac0000(0000)
knlGS:0000000000000000
[17243.492729] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17243.492748] CR2: ffffffffff600400 CR3: 0000003fdf1c8000 CR4:
00000000001407e0
[17243.492772] Stack:
[17243.492780]  ffffffff81d0ff80 ffff883fdd729a48 ffffffff81601c06
ffff880900000002
[17243.492812]  ffff883fdd729a32 ffff883f87759bc0 ffff883fdd729a94
ffff881e520368c0
[17243.492842]  ffffffff81d0ef40 ffff880063f05800 ffff883fdd729c60
ffff883fdd729a78
[17243.492872] Call Trace:
[17243.492885]  [<ffffffff81601c06>] xfrm_state_lookup+0x66/0x90
[17243.492907]  [<ffffffff8160796e>] xfrm_user_state_lookup+0x6e/0xe0
[17243.492930]  [<ffffffff81255078>] ? security_capable+0x18/0x20
[17243.492952]  [<ffffffff81608912>] xfrm_get_sa+0x42/0xc0
[17243.492971]  [<ffffffff816078db>] xfrm_user_rcv_msg+0x12b/0x150
[17243.493919]  [<ffffffff816077b0>] ? attach_auth+0xf0/0xf0
[17243.495228]  [<ffffffff815979ec>] netlink_rcv_skb+0x6c/0xd0
[17243.496520]  [<ffffffff81606e6f>] xfrm_netlink_rcv+0x3f/0x60
[17243.497813]  [<ffffffff8159779f>] netlink_unicast+0x12f/0x1e0
[17243.499162]  [<ffffffff815989b8>] netlink_sendmsg+0x348/0x3a0
[17243.500436]  [<ffffffff81551670>] sock_sendmsg+0x90/0xc0
[17243.501685]  [<ffffffff810def29>] ? futex_wait+0x1b9/0x290
[17243.502945]  [<ffffffff8154ee49>] ? copy_from_user+0x9/0x10
[17243.504203]  [<ffffffff815517a4>] SYSC_sendto+0x104/0x140
[17243.505466]  [<ffffffff811bdc73>] ? __sb_end_write+0x33/0x70
[17243.506713]  [<ffffffff815517ee>] SyS_sendto+0xe/0x10
[17243.507951]  [<ffffffff8163c8e9>] system_call_fastpath+0x16/0x1b
[17243.509368] Code: 14 41 31 da 41 31 c2 48 8b 87 20 0d 00 00 45 21 da
4a 8b 3c d0 31 c0 4c 8d 57 d8 48 85 ff 49 0f 45 c2 eb 74 0f 1f 80 00 00
00 00 <66> 44 39 88 c2 00 00 00 75 56 39 48 50 75 51 44 38 40 54 75 4b
[17243.512127] RIP  [<ffffffff81601b18>] __xfrm_state_lookup+0x78/0x100
[17243.513445]  RSP <ffff883fdd7299f0>


The objdump of the function:


00000000000030f0 <__xfrm_state_lookup>:
__xfrm_state_lookup():
.....
    3144:       48 8b 87 20 0d 00 00    mov    0xd20(%rdi),%rax
    314b:       45 21 da                and    %r11d,%r10d
    314e:       4a 8b 3c d0             mov    (%rax,%r10,8),%rdi
    3152:       31 c0                   xor    %eax,%eax
    3154:       4c 8d 57 d8             lea    -0x28(%rdi),%r10
    3158:       48 85 ff                test   %rdi,%rdi
    315b:       49 0f 45 c2             cmovne %r10,%rax
    315f:       eb 74                   jmp    31d5
<__xfrm_state_lookup+0xe5>
    3161:       0f 1f 80 00 00 00 00    nopl   0x0(%rax)
/root/zach/kernel/linux-3.14.13/net/xfrm/xfrm_state.c:676
    3168:       66 44 39 88 c2 00 00    cmp    %r9w,0xc2(%rax)
    316f:       00
    3170:       75 56                   jne    31c8
<__xfrm_state_lookup+0xd8>
    3172:       39 48 50                cmp    %ecx,0x50(%rax)
    3175:       75 51                   jne    31c8
<__xfrm_state_lookup+0xd8>
    3177:       44 38 40 54             cmp    %r8b,0x54(%rax)
    317b:       75 4b                   jne    31c8
<__xfrm_state_lookup+0xd8>


The failing offset is 0x3168, which is on line 676:

 667  static struct xfrm_state *__xfrm_state_lookup(struct net *net, u32
mark,
   668                                                const
xfrm_address_t *daddr,
   669                                                __be32 spi, u8 proto,
   670                                                unsigned short family)
   671  {
   672          unsigned int h = xfrm_spi_hash(net, daddr, spi, proto,
family);
   673          struct xfrm_state *x;
   674 
   675          hlist_for_each_entry(x, net->xfrm.state_byspi+h, byspi) {
   676                  if (x->props.family != family ||
   677                      x->id.spi       != spi ||
   678                      x->id.proto     != proto ||
   679                      !xfrm_addr_equal(&x->id.daddr, daddr, family))
   680                          continue;
   681 
   682                  if ((mark & x->mark.m) != x->mark.v)
   683                          continue;
   684                  xfrm_state_hold(x);
   685                  return x;
   686          }
   687 
   688          return NULL;
   689  }


This might be related to the other problem with excessive CPU usage of
charon in this mode and unnormal high number of SAs created- I will
write a separate thread about it.

Anyone seen this before?

Regards
Jiri Horky


More information about the Users mailing list