Prometheus scrape /metric with custom header

Question

I have an application that will be monitored by Prometheus, but the application need the custom header key like :

x-auth-token: <customrandomtoken>

What should I do with prometheus.yml?

Lumbago

Hi Ronaldo, check this Prometheus Documentation link: prometheus.io/docs/prometheus/latest/configuration/… Maybe the 'bearer_token' or http 'params' parameters it's what you need. yaml # Optional HTTP URL parameters. params: [ <string>: [<string>, ...] ] # Sets the `Authorization` header on every scrape request with # the configured bearer token. It is mutually exclusive with `bearer_token_file`. [ bearer_token: <secret> ] – Spool

i need custom header like ``` x-api-token: <mygeneratedcustomtoken> ``` – Lumbago

Then try using 'params:' to add a HTTP header – Spool

params is for "Optional HTTP URL parameters", it's definitely not a header. – Ketcham

Answer 1

Prometheus itself does not have a way to define custom headers in order to reach an exporter. The idea of adding the feature was discussed in this GitHub issue. Tl;dr: if you need a custom header, inject it with a forward proxy (I posted an example in another answer).

The prometheus-blackbox-exporter tag suggest that the question is about the exporter that makes probes, which is a separate thing and it does have a way to set headers. Only, it does not scrape metrics, it makes them.

Blackbox exporter has it's own configuration file and it consists of modules. A module is a set of parameters, defining how to do the probe and what result to expect. Here is an example of a module that looks for 200-299 response code and uses X-Auth-Token header:

modules:
  http_2xx_with_header:
    prober: http
    http:
      headers:
        X-Auth-Token: skdjfh98732hjf22exampletoken

More examples can be found here and the list of configuration options - here.

When you made the blackbox exporter to load the new configuration, you need to adjust Prometheus configuration as well:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx_with_header]  # <- Here goes the name of the new module
    static_configs:
      - targets:
        - http://prometheus.io
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Hafnium

Answer 2

Prometheus doesn't support specifying custom HTTP headers, which must be sent with each scrape request to scrape target :( However, it supports specifying Authorization header via authorization and basic_auth options at scrape_config section.

For example, the following config instructs Prometheus to send Authorization: My-Auth my-super-secret header with each scrape request to http://localhost:8428/metrics:

scrape_configs:
- job_name: foo
  authorization:
    type: "My-Auth"
    credentials: "my-super-secret"
  static_configs:
  - targets: ["localhost:8428"]

P.S. If you still need sending custom http headers with each request to remote target, then take a look at vmagent - monitoring agent, which understands Prometheus scrape configs and can scrape Prometheus targets. This is a project I work on. It provides additional features on top of standard scrape_configs from Prometheus, including headers option - see these docs. The headers option allows specifying arbitrary number of additional http headers, which need to be sent to scrape target. For example, the following config instructs sending x-auth-token: <customrandomtoken> http header with each request to http://foobar:1234:

scrape_configs:
- job_name: foo
  headers:
  - "x-auth-token: <customrandomtoken>"
  static_configs:
  - targets: ["foobar:1234"]

Cheesecloth