[strongSwan] Using xfrm marks to select right tunnels based on uid

[Jiri Horky]

Hi list,

I am wondering if it is possible to create ~1000 of tunnels from a single
linux machine for testing purposes and route the traffic based on UID of
the processes. I was trying to do PoC with just two tunnels  using the
following setup:

strongswan-5.5.3, kernel 4.12.5-gentoo

hotgeorge horky # cat /etc/ipsec.conf
conn node13
  auto=add
  type=tunnel
  keyexchange=ikev2
  ike=aes256-sha1-modp1024!
  esp=aes256-sha1-noesn!
  left=%defaultroute
  leftid="SomeID"
  leftsourceip=%config4
  right=node13
  rightid=myrightid
  rightsubnet=0.0.0.0/0
  authby=psk
  mark_out=13
  leftupdown=/usr/bin/sudo -E /etc/ipsec_mark_updown

conn node14
  auto=add
  type=tunnel
  keyexchange=ikev2
  ike=aes256-sha1-modp1024!
  esp=aes256-sha1-noesn!
  left=%defaultroute
  leftid="SomeID"
  leftsourceip=%config4
  right=node14
  rightid=myrightid
  rightsubnet=0.0.0.0/0
  authby=psk
  mark_out=14
  leftupdown=/usr/bin/sudo -E /etc/ipsec_mark_updown

I have set charon.install_routes = 0 to avoid installing default route to
route table 220, so I can actually setup the second tunnel.

I use following static iptables rules to mark the traffic based on UID:
iptables -t mangle -A OUTPUT -m owner --uid 1013 -j MARK --set-xmark
0xd/0xffffffff
iptables -t mangle -A OUTPUT -m owner --uid 1014 -j MARK --set-xmark
0xe/0xffffffff

The interesting content of /etc/ipsec_mark_updown script is the following:
MY_DEFAULT_GW=10.7.65.1
case $PLUTO_CONNECTION in
  *node13*)
  MARK=13
;;
  *node14*)
  MARK=14
;;
esac

ROUTE_TABLE=$((1000+MARK))

case $PLUTO_VERB in
  up-client)
ip route flush table $ROUTE_TABLE
ip rule del fwmark $MARK table $ROUTE_TABLE 2>/dev/null
ip rule add priority 10 fwmark $MARK table $ROUTE_TABLE
ip route add default via $MY_DEFAULT_GW proto static src $PLUTO_MY_SOURCEIP
table $ROUTE_TABLE
        ;;

Now, if I fire up the two tunnels:
ipsec up node13
ipsec up node14

I have the following routing rules:
hotgeorge horky # ip rule list
0: from all lookup local
10: from all fwmark 0xd lookup 1013
10: from all fwmark 0xe lookup 1014
220: from all lookup 220
32766: from all lookup main
32767: from all lookup default

And following routing table:
hotgeorge horky # ip route list table 1013
default via 10.7.65.1 dev wlo1  proto static  src 100.111.0.91
hotgeorge horky # ip route list table 1014
default via 10.7.65.1 dev wlo1  proto static  src 100.111.0.167
hotgeorge horky # ip route list table 220

Where table 220 is empty.

The trouble is the if I execute curl under user test_1013 (with uid 1013),
it times out on sending a DNS query:
su test_1013 -c "curl http://ip-info.ff.avast.com/v1/info"

When I manually specify the source address, it works:

su test_1013 -c "curl http://ipv4bot.whatismyipaddress.com --interface
100.111.0.91 --dns-ipv4-addr 100.111.0.91"; echo
77.234.40.153

su test_1014 -c "curl http://ipv4bot.whatismyipaddress.com --interface
100.111.0.167 --dns-ipv4-addr 100.111.0.167"; echo
77.234.40.182

So it seems that the kernel's source address selection does not work
correctly in my case. I am able to workaround my troubles by specifying
SNAT rules like:

iptables -t nat -A POSTROUTING -m mark --mark $MARK ! -s $PLUTO_MY_SOURCEIP
-j SNAT --to-source $PLUTO_MY_SOURCEIP

But I would like to avoid doing that.

Could you please enlighten me what I am doing wrong? It seems that if I let
strongwan install the routes to the table 220 (without any restrictions to
marks), the source address is selected correctly. This got me thinking that
the source ip selection probably happens sooner than marking the packets.
So I tried  to insert my marking rules based on uid to OUTPUT chain of raw
table. When I did this, it did not help and moreover, even the case with
manual selection of source ip in curl stopped working.

I would be grateful for any tips.

Note: I know that using network namespaces could sound like a better idea
than this. Unfortunatelly, even the newest kernel contain some race
conditions when destroying namespaces rapidly preventing us to use this way.

Thanks
Jiri Horky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.strongswan.org/pipermail/users/attachments/20171128/6a1982b3/attachment.html>