Prometheus node_exporter in docker: Host networking vs hostnames

To monitor a small home server, I run prometheus and node_exporter (and grafana, and several more things) on docker (similar to https://github.com/stefanprodan/dockprom). I run prometheus on a bridged docker network. For node_exporter, I have two options, which affect the node_network_transmit_bytes_total metric.

Using the same bridged docker network as prometheus
- Pro: nodeexporter can be addressed directly by name, thanks to docker's internal DNS
- Con: The node_network_transmit_bytes_total metric only has docker's virtual internal NIC, not the physical NIC of the box being monitored. This is depsite bind-mounting /proc from the host into /host/proc in the container (specifically my physical interface is eno0, visible in /proc/net/dev on the host):
```
$ docker exec -it nodeexporter2 cat /host/proc/net/dev | awk '{print $1}'
Inter-|
face
eth0:
lo:
```
Using host-mode networking for nodeexporter
- Pro: All NICs, including the physical host NIC, are visible
- Con: There does not appear to be a clean way for prometheus to address nodeexporter:
  - localhost means prometheus itself
  - Hostname of host seems inaccessible? Running docker exec -it prometheus wget -O - http://actual-hostname:9100/metrics works (and uses my host's LAN IP, 192.168.x.x), but configuring actual-hostname:9100 as a prometheus target gives an error (Get "http://actual-hostname:9100/metrics": dial tcp 127.0.1.1:9100: connect: connection refused). I'm not sure why they're resolving differently.
  - What I ended up doing is emulating the host.docker.internal feature available for docker-on-windows and docker-on-mac, by adding this to my docker-compose.yml:
```
extra_hosts:
- "host.docker.internal:172.18.0.1"
```
    That's very brittle, however: That 172.18 was just recently 172.19; I believe it changed on reboot or docker version upgrade. I'd love to do be able to set the extra_hosts to a result of running some script on the host to determine the correct network name, but that wouldn't automatically be re-run on boot.

Any advice?

Late edit: Thanks to thomas, turns out there's a magic host host-gateway that does this, so extra_hosts: ["host.docker.internal:host-gateway"] should do the trick. undocumented, but apparently it's implemented here. and already live in docker 20.10.6 (and likely earlier).

I ended up solving this by manually configuring the network:

networks:
  monitor-net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.23.0.0/16
          ip_range: 172.23.5.0/24
          gateway: 172.23.5.254

# ...

services:
  nodeexporter:
    network_mode: host
    # ...
  prometheus:
    networks:
      - monitor-net
    extra_hosts:
      - "host.docker.internal: 172.23.5.254"

Then prometheus has the target of host.docker.internal for node_exporter, and the address should be stable.

Recommended topics

Hot tags