How to list all producers of a kafka cluster?

Question

I am able to list all Kafka Consumer with KafkaAdminClient:

AdminClient client = AdminClient.create(conf);
ListTopicsResult ltr = client.listTopics();
KafkaFuture<Set<String>> names = ltr.names();

ArrayList<ConsumerGroupListing> consumerGroups = new ArrayList<>(client.listConsumerGroups().all().get());
ConsumerGroupListing consumerGroup = consumerGroups.get(0);

Is it possible to list all registrated producers in a similar way?

Judoka

Answer 1

In contrast to consumers, it is not possible to retrieve such information since Kafka brokers don't store any kind of information about the producers connected to them.

Epitome

Have you got a reference for this? – Frans

This is not true as the producers will be registered upon the first write and their ids expire by default 7 days after the last write. See docs.confluent.io/platform/current/installation/configuration/…. – Historical

Answer 2

It is possible to get all active producer using AdminClient API:

Collection<TopicPartition> topicPartitions = ...

DescribeProducersResult result = adminClient.describeProducers(
  Collections.singleton(topicPartition)
);

Map<TopicPartition, DescribeProducersResult.PartitionProducerState> producerStateMap = result.all().get();

https://kafka.apache.org/32/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html#describeProducers(java.util.Collection,org.apache.kafka.clients.admin.DescribeProducersOptions)

Vogel

This ultimately gives me a collection of ProducerStates, which contain a producerId as a long. However, I see no way to resolve that to a client.id or something similarly useful. – Apt