Extract TCP round trip time (RTT) estimations on linux

Question

I have apache server running on Ubuntu. Client connects and downloads an image. I need to extract RTT estimations for the underlying TCP connection. Is there a way to do this? Maybe something like running my tcp stack in debug mode to have it log this info somewhere?

Note that I don't want to run tcpdump and extract RTTs from the recorded trace! I need the TCP stack's RTT estimations (apparently this is part of the info you can get with TCP_INFO socket option). Basically need something like tcpprob (kprobe) to insert a hook and record the estimated RTT of the TCP connection on every incoming packet (or on every change).

UPDATE:

I found a solution. rtt, congestion window and more can be logged using tcpprobe. I posted an answer below.

Languish

#16232100 – Discountenance

Thanks! Yeah I'm familiar with "ss", but it's not exactly what I need as it will give me a very coarse sampling of the RTT. I need something like tcpprob (kprobe) to insert a hook and record the estimated RTT of the TCP connection on every incoming packet (or on every change). – Languish

Hmm. Well Linux congestion control algorithms are pluggable modules in the net/ipv4/ directory. Each computes rtt differently (and reno doesn't compute it at all!), so I'm thinking you'll have to hook into a particular implementation. I see you tagged cubic, so that's tcp_cubic.c. I don't think there's a generic method of getting the rtt, but I could certainly be wrong... – Discountenance

Thanks @Discountenance ! Yeah I haven't yet been able to find a generic way to do it, I probably need to get my hands dirty and do it myself. I am also experimenting with ss and see if it actually suffice for what I need. I'll keep you posted. – Languish

Oh boy! I can't believe tcpprobe was doing this for me the whole time and I missed it! haha I will post an answer in a bit – Languish

Have you tried this: https://mcmap.net/q/1167124/-how-to-set-get-socket-rtt-in-linux-socket-programming ? – Declarer

Interesting. No I have not, but will look into it. Thanks! – Languish

Answer 1

This can be done using tcpprobe, which is a module that inserts a hook into the tcp_recv processing path using kprobe records the state of a TCP connection in response to incoming packets.

Let's say you want to probe tcp connection on port 443, you need to do the following:

sudo modprobe tcp_probe port=443 full=1
sudo chmod 444 /proc/net/tcpprobe
cat /proc/net/tcpprobe > /tmp/output.out &
pid=$!

full=1: log on every ack packet received

full=0: log on only condo changes (if you use this your output might be empty)

Now pid is the process which is logging the probe. To stop, simply kill this process:

kill $pid

The format of output.out (according to the source at line 198):

[time][src][dst][length][snd_nxt][snd_una][snd_cwnd][ssthresh][snd_wnd][srtt][rcv_wnd]

Languish

The RTT is in "ms" – Languish

It doesn't seem to be available since at least 5.1.x. Do you know if it was removed or just renamed ? – Canicular

Hmm...unfortunately I don't know, as I haven't work with this for a while now. – Languish

Answer 2

This can be done without the need for any additional kernel modules using the ss command (part of the iproute package), which can provide detailed info on open sockets. It won't show it for every packet but most of this info is calculated over a number of packets. E.g. To list the currently open TCP (t option) sockets and associated internal TCP info (i) information - including congestion control algorithm, rtt, cwnd etc:

ss -ti

Here's some example output:

State       Recv-Q Send-Q        Local Address:Port        Peer Address:Port
ESTAB       0      0             192.168.56.102:ssh        192.168.56.1:46327
  cubic wscale:6,7 rto:201 rtt:0.242/0.055 ato:40 mss:1448 rcvmss:1392
advmss:1448 cwnd:10 bytes_acked:33169 bytes_received:6069 segs_out:134
segs_in:214 send 478.7Mbps lastsnd:5 lastrcv:6 lastack:5 
pacing_rate 955.4Mbps rcv_rtt:3 rcv_space:28960

Weywadt

ss documentation does not have any of those fields. E.g. what does "rtt: x/y" even mean? – Nb

For details on the fields take a look at the man page for ss linked to in the answer (Some distribution man pages seem to lack this info). For full details see the src code: git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/misc/… – Weywadt