Login/Register
How to add the Kafka Exporter as a data source to Grafana?
Asked Answered
Solved apache-kafkaprometheusgrafana
A

2

1

I'm trying a simplified example of using Kafka and the Kafka Exporter with Grafana. I have a docker-compose.yml similar to the following:

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    build: kafka-exporter
    ports:
      - "9308:9308"
    networks:
      - app-tier
    entrypoint: ["run.sh"]
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    networks:
      - app-tier

where run.sh is a wrapper script to keep retrying to run the exporter while Kafka is starting, see https://github.com/khpeek/kafka-exporter-example. The problem is that when I log into Grafana and try to add a Prometheus data source with URL http://kafka-exporter:9308/metrics, I get an error:

Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found <, error found in #1 byte of ...|<html> |..., bigger context ...|<html> <head><title>Kafka Exporter</title>|...

Here is how it looks in the UI: enter image description here

It seems like Grafana ignores the /metrics path and tries to scrape data from http://kafka-exporter:9308 directly, which indeed looks like the error message describes:

enter image description here

By contrast, the /metrics endpoint contains the actual metrics:

> curl localhost:9308/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 20
...

Why am I getting this error? Shouldn't Grafana pick up the path?

Anticlimax answered 5/1, 2022 at 1:40 Comment(0)
S
3

You cannot connect Grafana Prometheus datasource to an exporter directly. You need to set up a Prometheus server, scrape from that server the kafka exporter metrics and finally connect Grafana to the Prometheus server.

Syracuse answered 5/1, 2022 at 7:14 Comment(0)
A
4

To show the implementation of @usuario's answer, the docker-compose.yml needs a Prometheus service,

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    build: kafka-exporter
    ports:
      - "9308:9308"
    networks:
      - app-tier
    entrypoint: ["run.sh"]
  cli:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  prometheus:
    image: bitnami/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml"
    networks:
      - app-tier
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    networks:
      - app-tier

where the prometheus/prometheus.yml configuration file is statically configured to scrape the Kafka Exporter:

global:
  scrape_interval: 10s
  scrape_timeout: 10s
  evaluation_interval: 1m

scrape_configs:
  - job_name: kafka-exporter
    metrics_path: /metrics
    honor_labels: false
    honor_timestamps: true
    sample_limit: 0
    static_configs:
      - targets: ['kafka-exporter:9308']

Now the Prometheus data source can be added and metrics from the Kafka Exporter such as kafka_consumergroup_lag can be viewed:

enter image description here

Anticlimax answered 5/1, 2022 at 21:36 Comment(0)
S
3

You cannot connect Grafana Prometheus datasource to an exporter directly. You need to set up a Prometheus server, scrape from that server the kafka exporter metrics and finally connect Grafana to the Prometheus server.

Syracuse answered 5/1, 2022 at 7:14 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.