Extract unique IPs from live tcpdump capture

Question

I am using the following command to output IPs from live tcpdump capture

sudo tcpdump -nn -q ip -l | awk '{print $3; fflush(stdout)}' >> ips.txt

I get the following output

192.168.0.100.50771
192.168.0.100.50770
192.168.0.100.50759

Need 2 things:

1. Extract only the IPs, not the ports.
2. Generate a file with unique IPs, no duplicated, and sorted if posible.

Thank you in advance

Landbert

Getting $3 from that output gets you the source IP, which may not be present on your network; it may be a packet sent FROM somewhere and eventually routed into your network. There's no way to be 100% sure which IPs are local - and therefore whether to use the source or destination address from the tcpdump output - except packets where the source or destination are in the same network as your machine. If that's a given, you'd have to code that check (is IP in the same subnet as one of my IPs) into the awk script, and THEN start worrying about parsing away the ports, and uniqueness. – Dune

Answer 1

To extract unique IPs from tcpdump you can use:

awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+).*/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' YOURFILE

So your command to see unique IPs live would be:

sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }'

This will print each IP to output as soon as they appear, so it cannot sort them. If you want to sort those, you can save the output to a file and then use sort tool:

sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' > IPFILE
sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4  IPFILE

Example output:

34.216.156.21
95.46.98.113
117.18.237.29
151.101.65.69
192.168.1.101
192.168.1.102
193.239.68.8
193.239.71.100
202.96.134.133

NOTE: make sure you are using gawk. It doesn't work with mawk.

Lepage

Excellent answer! In this part of the code if(!d[ip]) could I add a awk condition to check if the IP is public or private? In order to output only public IP addresses. Using grep would be something like grep -E '^(192\.168|10\.|172\.1[6789]\.|172\.2[0-9]\.|172\.3[01]\.)' but how can be done using awk? – Landbert

Sure, just add $0!~/^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ at the start of your AWK script for non-local IPs and $0~/^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ for local IPs. So the start of AWK command would be like: awk '$0~/^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ { ip = ... }' – Lepage

sudo tcpdump -nn -q ip -l | gawk '$0!~/^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ { ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' outputs every IP without filtering. Can you help me please? – Landbert

Oh, sorry, you need to change $0 to $3. – Lepage

@AndriyMakukha : Could you please explain how are only unique IPs captured? it has to do something with d[IP] but I am having a hard time understanding it. thanks! – Tem

@aj8080, d[IP]=1 will add pair (IP, 1) into the associative array d (also known as "dictionary" in some languages). So condition if (!d[IP]) will prevent any previously observed IPs from being processed. – Lepage

Answer 2

While I'm a huge Awk fan, it's worthwhile having alternatives. Consider this example using cut:

  tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | sort | uniq

Hanahanae

Your command does not output any IP addresses. Here is a solution based on yours: sudo tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | awk '!x[$0]++' – Charcot

Answer 3

This is a using match (working in macOs)

sudo tcpdump -nn -q ip -l | \
    awk '{match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
    ip = substr($3,RSTART,RLENGTH); \
    if (!seen[ip]++) print ip }'

In case want to pre-filter the input you could use something like:

sudo tcpdump -nn -q ip -l | \
    awk '$3 !~ /^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ \
    {match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
    ip = substr($3,RSTART,RLENGTH); \
    if (!seen[ip]++) print ip }'

Jabin

Answer 4

sudo tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | awk '!x[$0]++'

Is the command that did it for me. Simple and elegant.

Charcot

While this works, it takes around a minute to get output. Is there a way to do this but in real time stream? – Forney