[strongSwan] traffic analysis in the server network - how to do it easiest?

[Andreas Schuldei]

for almost a year now we have the infrastructure (configuration wise)
to be able to run ipsec between our servers in a full host-to-host
mash.

one important puzzle bit is the smooth deployment, though. it is
necessary to deploy it in small, controlled steps, and it is important
to be able to roll back each deployment if stuff does not work out as
planned.

for that we need to know pretty much exactly what our traffic looks
like and what servers talk to which others within our network (of
debian servers, only). so i need to collect data on which servers talk
to which other on which port and what protocol (udp, tcp, icmp, ...).
volume and time is not so very important to start with. later on i
want to be able to categorize the traffic patterns and be able to pick
first harmless and non-critical traffic and later on more and more
important ones. it would also be very useful to recognize and
understand traffic patterns like "an AP talks to these other server
classes on these ports, but never ever with these others".

how can i collect this data without creating unnecessary load on the
servers? Are there e.g. some smart iptables rules/counters that i can
use and make regular snapshots of? Because I have only a vague idea
about what the traffic looks like i would like to count/measure every
sort of traffic. and how can i analyze and categorize the resulting
data in an meaningful way? I am willing to do some coding, but would
appreciate some (voluntary) help. this is a cool project i am
interested in (which even has real business value!) but I never seem
to get around to do it. i have a strong hunch that there are cool
mathematical models out there for precisely this, but i don't even
know what i am looking for and need hints and/or help.

/andreas