How I get the number of (currently) established TCP connections in prometheus (kubernetes monitoring)

Question

I am using this command in linux to see (currently) established TCP connections:

netstat -ant | grep ESTABLISHED | wc -l 

How can i translate this command to PromQL (per node) ?

I am using prometheus with node exporter in my kubernetes cluster

Toaster

Answer 1

To get number of currently open TCP connections, you can use node_netstat_Tcp_CurrEstab (Gauge) metric.

you can also use node_netstat_Tcp_ActiveOpens (Counter) metrics with appropriate rate such as

rate(node_netstat_Tcp_ActiveOpens[10m])

These metrics are based on TCP-MIB (RFC-4022) and they are obtained by parsing /proc/net/netstat and /proc/net/tcp files on every node running exporter.

Somewise

Thanks, Can i also use /proc/net/sockstat and /proc/net/sockstat to get established connections (inuse), connections in a time wait state (tw) and total tcp connections (alloc) ? – Toaster

@Toaster yes, just find out metrics name based on mentioned MIB – Somewise

i have a pb where node_netstat_Tcp_CurrEstab in prometheus give only 2 but when i do netstat -ant | grep ESTABLISHED | wc -l, i am getting 3000. any idea ? – Toaster

well, that's general problem with Gauge metrics such as node_netstat_Tcp_CurrEstab. You will get only value that Prometheus asked for (usually every 30sec). So that's exact situation where counter metric (such as node_netstat_Tcp_ActiveOpens) comes handy. If you plot it in graph, you will got increase over time and don't miss moment where process opened bunch of connections just after scrape of metric was done – Somewise

I Did this post for this pb with the graph from grafana #68820167 – Toaster