Docker - Prometheus container dies immediately

Question

I have cadvisor running with port mapping 4000:8080 and I have to link it with a container with prometheus.

My prometheus.yml is:

scrape_configs:
# Scrape Prometheus itself every 2 seconds.
- job_name: 'prometheus'
  scrape_interval: 2s
  target_groups:
  - targets: ['localhost:9090', 'cadvisor:8080']

This file has path /home/test/prometheus.yml. To run the container with prometheus, I do:

docker run -d -p 42047:9090  --name=prometheus -v /home/test/prometheus.yml:/etc/prometheus/prometheus.yml  --link cadvisor:cadvisor prom/prometheus -config.file=/etc/prometheus/prometheus.yml -storage.local.path=/prometheus -storage.local.memory-chunks=10000 

The container is created, but it dies immediately. Can you tell me where the problem is?

Messages form docker events& :

2016-11-21T11:43:04.922819454+01:00 container start 69d03c68525c5955cc40757dc973073403b13fdd41c7533f43b7238191088a25 (image=prom/prometheus, name=prometheus)
2016-11-21T11:43:05.152141981+01:00 container die 69d03c68525c5955cc40757dc973073403b13fdd41c7533f43b7238191088a25 (exitCode=1, image=prom/prometheus, name=prometheus)

Hypsometer

Answer 1

Config format is changed. targets come under static_config in the latest version.

scrape_configs:
# Scrape Prometheus itself every 2 seconds.
  - job_name: 'prometheus'
  scrape_interval: 2s
  static_configs:
      - targets: ['localhost:9090', 'cadvisor:8080']

Prometheus Documentation for further help

Triparted

Answer 2

Yes, target_groups is renamed to static_configs. Please use the latest Prometheus image with the following.

static_configs:
  - targets: ['localhost:9090', 'cadvisor:8080']

The above worked for me.

Modigliani

Answer 3

I think target_groups have been deprecated from scrape_configs in latest version of prometheus. you can try static_configs or file_sd_config

scrape_config
static_config
file_sd_config

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets:
         - "stg-elk-app-01:9100"
         - "stg-app-02:9100"

Corncob

Answer 4

The indentation isn't correct, try:

scrape_configs:
  # Scrape Prometheus itself every 2 seconds.
  - job_name: 'prometheus'
    scrape_interval: 2s
    target_groups:
    - targets: ['localhost:9090', 'cadvisor:8080']

Translocate

Which version of Prometheus are you using? target_groups was renamed to static_configs a while back. This is very difficult to debug without any error output. – Translocate

I have done "docker pull prom/prometheus" , so I think it should be the latest (the TAG of the image is ''latest'') – Hypsometer

from logs: time="2016-11-21T11:21:40Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in scrape_config: target_groups" source="main.go:149" – Hypsometer

Answer 5

As you said in your earlier comment:

from logs: time="2016-11-21T11:21:40Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in scrape_config: target_groups" source="main.go:149"

Which clearly means that the field "target_groups" is causing the problem. This is due to the fact that the new version of Prometheus (v1.5 onwards) have discarded the use of "target_groups" field and simply provide the targets. Even I faced this issue about 6 months ago. Please try it with a new version. The docker pull prom/prometheus might be getting you the old one.

Hope this helps...!!!

Salome

Answer 6

The name of the container is prometheus.

Generally, when a container exists immediately after it starts, I would recommend adding the -log.level=debug right after -config.file.

docker run -d -p 42047:9090 --name=prometheus -v /home/test/prometheus.yml:/etc/prometheus/prometheus.yml --link cadvisor:cadvisor prom/prometheus -config.file=/etc/prometheus/prometheus.yml -log.level=debug -storage.local.path=/prometheus -storage.local.memory-chunks=10000

Next, see the logs for the container:

docker logs prometheus

Any issues with the configuration will be there.

Analysand