Prometheus blackbox-exporter with endpoint authentication?

Question

I am pretty new to the Prometheus and not sure how I am going to ping the endpoint with authentication. Not sure my problem could be solved by built-in prometheus configs, let me describe the flow I would like to achieve:

(1) send http POST with {username, password} to api endpoint company.com/auth

(2) should retrieve {access_tokens, refresh_token...} of Bearer token type

(3) preserve this access_token and direct to other pages. All the other endpoints should only be pinged, if the access_token is already there and correct.

(4)After authentication, it should still send the http request at some frequency and output the metrics just as the blackbox-exporter do.

Basically I am trying to mimic the same procedure for sequence of API calls in Postman tests. I've seen config of blackbox-exporter of basic_auth and bearer_token, but not sure how to actually setup the params and how to redirect to other pages.

(Should I set the basic_auth username and password according to (1)? Where the token is returned? And should I replace the token to bearer_token?)

Any guidance on this would be great! I am pretty new to the whole process, sorry if the question is too fundamental or ambiguous. Thanks in advance and really appreciate any help!

Smithery

Answer 1

Step 2/3 requires running Javascript, which the Blackbox exporter cannot do.

The Blackbox exporter will follow redirects automatically, use URL parameters, and send bearer tokens but you need to know what you want to send in advance. You might have some luck with https://github.com/mattbostock/webdriver_exporter, you could write your own exporter, or look for a simpler blackbox test that the blackbox exporter can execute.

Hamish

Thanks for the clarification! I ended up wrote a automation tool to scrap all the data, but we still use prometheus for other resource monitoring. It's really powerful! – Smithery

Answer 2

How to monitor authenticated API endpoint that requires username and password using Blackbox exporter.

Edit the blackbox.yml

modules:
  http_2xx:    # Prometheus.yml file
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET # post ,put ,delete

  http_2xx_auth:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: true
      basic_auth:
        username: "username"
        password: "password"

Edit the prometheus.yml

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://ww.xyx.com
        - https://app.pqs.cloud
        - https://app.abc.cloud/actuator/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
  - job_name: 'blackbox_auth_based'
    metrics_path: /probe
    params:
      module: [http_2xx_auth]
    static_configs:
      - targets:
        - https://api.auth.com/actuator/health   # It is authenticated by username and password
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # The blackbox exporter's real hostname:port.

Restart the services prometheus and blackbox

sudo systemctl restart prometheus.service
sudo systemctl restart blackbox.service

https://medium.com/@sheikhaafaq18/monitoring-of-open-and-secure-api-endpoints-using-prometheus-blackbox-dcc2eabebf93

Urticaria

Answer 3

Not sure whether you have still the same issue but i have tried to setup the api checks (whether api is working fine or not) via blackbox exporter and while installing the exporter i have used below config.

config:
  modules:
    http_2xx:
      prober: http
      timeout: 5s
      http:
        valid_http_versions: ["HTTP/1.1", "HTTP/2"]
        no_follow_redirects: false
        preferred_ip_protocol: "ip4"
    api_checks:
      prober: http
      timeout: 5s
      http:
        method: GET
        valid_http_versions: ["HTTP/1.1", "HTTP/2"]
        no_follow_redirects: false
        preferred_ip_protocol: "ip4"
        bearer_token_file: /var/secret.file

First module is for checking the normal endpoint whether they are reachable or not. AND second module is for checking the api with a bearer token. I have created a secret and mounted it as volume on to the pod at /var/secret.file

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: bearer-token
data:
  secret.file: |
    Z2hwX1lvbsyYXVKeTFndlQzRQo=

and added those to deployment file as below.

spec:
  templates:
    metadata:
       labels:
         app: xxxx
    spec:
      containers:
      - name: XXXX
        image: "image-name" 
        volumeMounts:
          - name: bearer-token
            mountPath: /var/
      volumes:
      - name: bearer-token
        secret:
          secretName: bearer-token

After that we need to go to prometheus setup and add a scrape config for the same as below

prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml
      - /etc/config/alerting_rules.yml
    ## Below two files are DEPRECATED will be removed from this default values file
      - /etc/config/rules
      - /etc/config/alerts

    scrape_configs:
      - job_name: prometheus
        static_configs:
          - targets:
            - localhost:9090

      - job_name: Blackbox-check
        metrics_path: /probe
        static_configs:
          - targets:
            - www.google.com/
            - https://api.github.com/user/repos
        relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-prometheus-blackbox-exporter:9115

      - job_name: api_checks
        metrics_path: /probe
        params:
          module: [api_checks]
        static_configs:
          - targets:
            - https://api.github.com/user/repos
        relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-prometheus-blackbox-exporter:9115

It will work for you.

Sitin