prometheus dynamic metrics_path

Question

Prometheus allows me to dynamically load targets with file_sd_config from a .json file like this

#prometheus.yaml
- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'

[
  {
    "labels": {
      "job": "kube-metrics"
    },
    "targets": [
      "http://node1:8080",
      "http://node2:8080"
    ]
  }
]

However my targets differ in the metrics_path and not the host (I want to scrape metrics for every kubernetes node on <kube-api-server>/api/v1/nodes/<node-name>/proxy/metrics/cadvisor) but I can only set the metrics_path at the job level and not per target. Is this even achievable with prometheus or do I have to write my own code to scrape all these metrics and export them at a single target. Also I couldn't find a list of all supported auto discovery mechanisms, did I miss something in the docs?

Refulgent

Answer 1

You can use relabel_config in Prometheus config to change __metrics_path__ label config.

The principe is to provide the metrics path in your targets under the form host:port/path/of/metrics (note: drop the http://, it is in scheme parameter of scrape_config)

[
  {
    "targets": [
      "node1:8080/first-metrics",
      "node2:8080/second-metrics"
    ]
  }
]

And then replace the related meta-labels with the parts

- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'
  relabel_configs:
    - source_labels: [__address__]
      regex:  '[^/]+(/.*)'            # capture '/...' part
      target_label: __metrics_path__  # change metrics path
    - source_labels: [__address__]
      regex:  '([^/]+)/.*'            # capture host:port
      target_label: __address__       # change target

You can reuse this method on any label known at configuration time to modify the config of the scrape.

On Prometheus, use the service discovery page to check your config has been correctly modified.

The official list of service discovery is in the configuration documentation: look for the *_sd_config in the index.

Concern

Answer 2

There is a slightly more elegant way of doing this, that doesn't involve manipulating the address. You can have a label with your target that can be the source of your "relabel" action, something like this:

- labels:
    __meta_discovery_path: '/first-metrics'
  targets:
  - 'node1:8080'
- labels:
    __meta_discovery_path: '/second-metrics'
  targets:
  - 'node2:8080'

And then your relabel would simply be:

  relabel_configs:
    - source_labels: [__meta_discovery_path]
      target_label: __metrics_path__  # change metrics path

Since the label being used starts with "__", it will be stripped from the metrics being pulled, making it nice and clean.

Custos

Adding the label __metrics_path__ to the service discovery's scrape config would suffice. This way you don't need to relabel it. – Selfforgetful

Answer 3

I wanted something like the below endpoints and achieved with this config

192.168.1.1:80/metrics/my-path1
192.168.1.2:80/metrics/my-path1

This will replace $1 with _new_path value.

- job_name: prometheus_dynamic_metrics_path
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  relabel_configs:
  - source_labels: [_new_path]
    separator: ;
    regex: (.*)
    target_label: __metrics_path__
    replacement: /metrics/$1
    action: replace
  static_configs:
  - targets:
    - 192.168.1.1:80
    labels:
      _new_path: my-path1
  - targets:
    - 192.168.1.2:80
    labels:
      _new_path: my-path2

Agiotage

Answer 4

You can set metrics path and scheme at target level using special labels. You can even change the scrape interval, timeout and job name:

[
  {
    "targets": [ "yourhost:1234" ],
    "labels": {
      "job": "override job's name",
      "__metrics_path__": "/my/custom/path",
      "__scheme__": "https",
      "__scrape_interval__": "30s",
      "__scrape_timeout__": "30s"
    }
  }
]

Selfforgetful