How to expose kafka metrics to /actuator/metrics with spring boot 2

Question

I was looking a while and didn't seem to find the answer. I'm using Spring boot 2, Spring Kafka 2.1.4 and I want to see the kafka consumer metrics in the /metrics endpoint of spring boot actuator. What I don't understand is - should I implenemt the exposure myself or is this comes out-of-box in boot 2 ?

If I to implement this my self, what is the best way to do it?

Hegemony

Kafka exposes almost the same metrics through JMX, so I suggest you to create a custom MeterBinder to expose these metrics. Plus, you can create spring boot autoconfigure and starter modules to simplify the usage. You can find an example (ready to use) here: github.com/luiztoscano/spring-boot-kmetrics Hope it helps. – Forbidding

Answer 1

Targeted for micrometer v1.1.0 is the KafkaConsumerMetrics implementation of MeterBinder. This should expose the kafka consumer metrics you're looking for.

Source Reference:

https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaConsumerMetrics.java

Detour

Is there an equivalent KafkaProducerMetrics available? – Recess

it is deprecated since v1.4.0 see KafkaClientMetrics – Arguelles

Answer 2

add this config works for me:

spring.jmx.enabled=true

Proceed

This might be the correct answer for the time of writing this. It worked for me. – Gnomic

@Proceed - Can you also put what other changes you did? Was there any code change required or Just actuator dependency jar, prometheus jar and spring.jmx.enabled=true Kindly provide full details how you have solved – Glottic

Answer 3

You are right, there is no out-of-the-box one. You don't have choice unless to implement your own MeterBinder. For the Apache Kafka Consumer metrics per se, you should inject a KafkaListenerEndpointRegistry and call its getListenerContainers() and use their metrics() to bind to the provided MeterRegistry.

When you come up with something, feel free to contribute the solution back to Spring Boot.

Spithead

The problem seems to be that there is no elegant way to migrate the KafkaMetric objects to Micrometer metrics. The only way I see is looping all kafka metrics and creating Micrometer identical metric for each of them – Hegemony

@MichaelAzimov I am looking to move the KafkaStreams metrics to micrometer. How did you proceed for the conversion ? – Briticism

Answer 4

This works for me(For Spring Boot 2.3.0.RELEASE and above):

@Autowired
private MeterRegistry meterRegistry;

@Bean
public ConsumerFactory<Object, Object> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, JsonDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);

    //More code here

    DefaultKafkaConsumerFactory<Object,Object> cf = new DefaultKafkaConsumerFactory<>(props);
    cf.addListener(new MicrometerConsumerListener<>(meterRegistry));
        
    return new DefaultKafkaConsumerFactory<>(props);
}

build.gradle
implementation 'io.micrometer:micrometer-registry-prometheus'

Metrics will be available at /actuator/prometheus/kafka_consumer_<metric-name>.

Winne