[strongSwan] Problem with MTU in IPSec Transport Mode

[Jan-Philipp Hülshoff]

Hello,
I have the following setup:

A private network "home" with a router "router" doing NAT with one 
static public ip. The router establishes a secure IPSec transport from 
his public ip to a server "server" on the internet (same as in example [1]).

The following problem can be observed:
Packets sent from inside the home network that are larger than some 
value and have the don't fragment bit set get lost somewhere in the 
router. No encrypted packets are then sent to the server. The router 
does not send ICMP host unreachable fragmentation needed messages back 
to the client in the home network. (I suspect that the resulting 
encrypted packet will be larger than the mtu of the outgoing interface.)

for example this affects ssh from inside the home network to server.


Is this behaviour intended? Is this use case supported or is it an 
unusual way to use ipsec transport mode in combination with NAT/routing?


Here are the tests that demonstrate the behaviour:
client-in-home# ping -s 1450 -M do server
PING server (IP) 1450(1478) bytes of data.
^C
--- server ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

As you can see here, the ping packets get lost. I can confirm that the 
router does not send any packets to the server.

client-in-home# ping server -s 1430 -M do
PING server (IP) 1430(1458) bytes of data.
1438 bytes from server (IP): icmp_seq=1 ttl=50 time=17.8 ms
1438 bytes from server (IP): icmp_seq=2 ttl=50 time=16.2 ms
1438 bytes from server (IP): icmp_seq=3 ttl=50 time=16.3 ms
1438 bytes from server (IP): icmp_seq=4 ttl=50 time=18.2 ms
^C
--- server ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 16.205/17.146/18.236/0.912 ms

But sending smaller packets works without problems. On the router the 
same ping command works as expected and gives the following output:

router# ping -s 1450 -M do server
PING server (IP) 1450(1478) bytes of data.
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long
^C
--- server ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2005ms

As you can see the router does not send an icmp fragmentation needed 
message to client-in-home.
I can confirm this behaviour with another installation with 3 vm's. I 
used Debian GNU/Linux jessie (kernel 3.16.0-4-amd64, strongswan 
5.2.1-6+deb8).

Kind regards,
  Jan-Philipp Hülshoff

[1]: 
https://www.strongswan.org/testing/testresults/ikev2/host2host-transport/