Docker stat network traffic

Question

I want ask 2 question about docker stats

for example

NAME                              CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
container_1                       1.52%               11.72MiB / 7.388GiB   0.15%               2.99GB / 372MB      9.4MB / 0B          9

in this situation net i/o statement 2.99GB / 372MB how much time reflected in that?

for one hour? or all of time?

and how can check docker container network traffic for an hour or minute?

i would appreciate if you any other advice. thank you

Damascene

Answer 1

This blog explains the network io of the docker stats command

Displays total bytes received (RX) and transmitted (TX).

If you need finer grained access, the blog also suggests to use the network pseudo files on your host system.

$ CONTAINER_PID=`docker inspect -f '{{ .State.Pid }}' $CONTAINER_ID`
$ cat /proc/$CONTAINER_PID/net/dev

---

To your second part: I'm not aware of any build-in method to get the traffic over the specific period, others might correct me. I think the easiest solution is to poll one of the two interfaces and calculate the differences yourself.

Folks

Answer 2

Docker documentation has pretty good description of collected metrics:

Since each container has a virtual Ethernet interface, you might want to check directly the TX and RX counters of this interface. Each container is associated to a virtual Ethernet interface in your host, with a name like vethKk8Zqi. Figuring out which interface corresponds to which container is, unfortunately, difficult.

You can obtain the statistics using:

TASKS=/sys/fs/cgroup/devices/docker/$CID*/tasks
PID=$(head -n 1 $TASKS)
mkdir -p /var/run/netns
ln -sf /proc/$PID/ns/net /var/run/netns/$CID
ip netns exec $CID netstat -i

When you check the moby source code and the underlying netns library it seems to be using the same approach to obtain metrics.

Once you have the symlink, you can obtain basic interface stats:

$ ip netns exec $CID netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500 1143027778      0      0 0      954730124      0      0      0 BMRU
lo       65536   740216      0      0 0        740216      0      0      0 LRU

or use netstat -s to obtain more detailed statistics:

$ ip netns exec $CID netstat -s
Ip:
    Forwarding: 1
    1143698426 total packets received
    0 forwarded
    0 incoming packets discarded
    1143698426 incoming packets delivered
    955432808 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    188729 active connection openings
    61742 passive connection openings
    10 failed connection attempts
    1171 connection resets received
    104 connections established
    1143682773 segments received
    1012787644 segments sent out
    35385 segments retransmitted
    0 bad segments received
    28045 resets sent
Udp:
    15653 packets received
    0 packets to unknown port received
    0 packet receive errors
    15663 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    4654 packets pruned from receive queue because of socket buffer overrun
    89540 TCP sockets finished time wait in fast timer
    121 packetes rejected in established connections because of timestamp
    13364331 delayed acks sent
    29007 delayed acks further delayed because of locked socket
    Quick ack mode was activated 576996 times
    766906525 packet headers predicted
    4131271 acknowledgments not containing data payload received
    170286131 predicted acknowledgments
    TCPSackRecovery: 9134
    Detected reordering 42755 times using SACK
    Detected reordering 1554 times using time stamp
    667 congestion windows fully recovered without slow start
    1319 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 1511
    263 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 5413
    TCPSackFailures: 38
    6 timeouts in loss state
...

or use more modern nstat:

$ ip netns exec $CID nstat -a
#kernel
IpInReceives                    1143922124         0.0
IpInDelivers                    1143922124         0.0
IpOutRequests                   955622552          0.0
TcpActiveOpens                  188757             0.0
TcpPassiveOpens                 61756              0.0
TcpAttemptFails                 10                 0.0
TcpEstabResets                  1171               0.0
TcpInSegs                       1143906468         0.0
TcpOutSegs                      1012988866         0.0
TcpRetransSegs                  35388              0.0
TcpOutRsts                      28050              0.0
UdpInDatagrams                  15656              0.0
UdpOutDatagrams                 15666              0.0
TcpExtPruneCalled               4654               0.0
TcpExtTW                        89559              0.0
TcpExtPAWSEstab                 121                0.0
TcpExtDelayedACKs               13367159           0.0
TcpExtDelayedACKLocked          29015              0.0
TcpExtDelayedACKLost            577079             0.0
TcpExtTCPHPHits                 767050142          0.0
TcpExtTCPPureAcks               4132020            0.0
TcpExtTCPHPAcks                 170323527          0.0
TcpExtTCPSackRecovery           9136               0.0
TcpExtTCPSACKReorder            42764              0.0
TcpExtTCPTSReorder              1555               0.0
TcpExtTCPFullUndo               668                0.0
TcpExtTCPPartialUndo            1320               0.0
TcpExtTCPDSACKUndo              1511               0.0

these are all incrementing counters, you would need a snapshot and compute a diff over some period, watch can give you some idea how fast are the metrics changing:

watch -d -n1 ip netns exec $CID nstat -a

Lambertson