Can't collect docker metrics using prometheus

Question

I want to monitor the Docker engine. I am following "Collect Docker metrics with Prometheus" to configure the Docker daemon for metrics which can be later collected and analyzed with Prometheus.

On my PC (OS version is CentOS 7.4, Docker version is 17.12.0.ce), I have pasted the following to /etc/docker/daemon.json

    {
      "metrics-addr" : "0.0.0.0:9323",
      "experimental" : true
    }

Then, starting the Prometheus instance, I found the connection between Prometheus and Docker daemon is failed.

Error log

Get http://localhost:9323/metrics: dial tcp 127.0.0.1:9323: connect: connection refused

What can I do next?

Coraleecoralie

what is your prometheus.yaml file? – Gamp

- job_name: 'docker' static_configs: - targets: ['localhost:9323'] – Coraleecoralie

did you tried "metrics-addr" : "127.0.0.1:9323" ? – Gamp

yes, i tried, but failed. i try to use curl localhost:9323/metrics on local machine, it's OK. but prometheus instance still can't connect to docker. – Coraleecoralie

Answer 1

In respect to your question

What can I do next?

after changing the configuration in /etc/docker/daemon.json it will be necessary to

systemctl daemon-reload
systemctl restart docker 

You may check after this if the port 9323 is in state LISTEN

lsof -Pi TCP -a -c dockerd 

As you mentioned in your comment that

curl http://$(hostname):9323/metrics  

is working properly on your local host, it indicates a problem with(in) your network.

As the Prometheus service is usually running somewhere else you may check if there is a connection from the remote machine to your Docker host. For this you could use something like

root@prometheusHost:/# nc -vz dockerHost 9323

It will give you a hint if the connection is refused, e.g. by a firewall.

Object

Reloading the daemon is what made it worked for me without anything further. – Tonettetoney

What if docker is on host machine, but prometheus inside container? – Ferne

Answer 2

Get docker ip addres

ip addr show docker0    

Enter docker ip addres in prometherus.yml configuration

    static_configs:
  - targets: ['172.17.0.1:9323']

Entrant

It works. thanks. Can you explain why 127.0.0.1 and 0.0.0.0 don't work – Conduplicate

@haj_baba: probably because prometheus is inside container, but 9323 port is on host machine. – Ferne

Answer 3

Configure the Docker daemon

NOTE: 0.0.0.0 i.e. all IP addresses on the local machine

/etc/docker/daemon.json

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Open firewall port 9323 --permanently

$ sudo firewall-cmd --zone=public --permanent --add-port=9323/tcp
success

A few checks

Check, remember to list --permanent configuration, or it won't show

$ sudo firewall-cmd --zone=public --permanent --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client http https ssh
  ports: 9323/tcp                            <= OPEN
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

$ sudo lsof -Pi TCP -a -c dockerd
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
dockerd-c 28157 root   14u  IPv6 30732963      0t0  TCP *:9323 (LISTEN)
...

Check connection to localhost

nc -vz localhost 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to ::1:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check connection to host IP e.g. 10.223.37.14

$ nc -vz 10.223.37.14 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.223.37.14:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check the metrics acccess using curl

curl http://localhost:9323/metrics

or using the IP from another machine

curl http://10.223.37.14:9323/metrics

You will get a connection refused if the port is not open

curl http://10.223.37.14:9323/metrics
curl: (7) Failed to connect to 10.223.37.14 port 9323: Connection refused

but once it's open, you will be able to see the metrics

$ curl http://10.223.37.14:9323/metrics
# HELP engine_daemon_container_actions_seconds The number of seconds it takes to process each container action
# TYPE engine_daemon_container_actions_seconds histogram
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.005"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.01"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.025"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.05"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.1"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.25"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.5"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="1"} 1
...

Faena

Answer 4

If you run Prometheus from a container, you can consider the below address:

host.docker.internal:9323

Add this to your prometheus.yml inside your Prometheus container.

Tews

Answer 5

I've run into the same issue, here's the fix: $ sudo vi /etc/hosts

0.0.0.0 <hostname> host

0.0.0.0 can be substituted with the advertised address in Docker Swarm

Sumer