How to find which consumer is assigned to which partition of a topic in kafka?

Question

I am building a kafka manager tool and I need to check which topic-partition is assigned to which consumer in a consumer group.

Suppose there is consumer-Group group-A consuming topic topic-A with n partitions, so there can be multiple consumers in group-A hosted in different VM's. So how to find which partitioned is assigned to which consumer host? Is it possible in kafka 0.9.1?

Thanks in advance.

Auerbach

Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data. – Ceremonious

Answer 1

You can check how $KAFKA_HOME/bin/kafka-consumer-groups.sh works and integrate its implementation into your kafka manager tool, this tool will show you detailed group owner information(for example, partition assignment, lag, IP).

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
page_visits_10k 0 500 3333 2833 consumer-1_/10.139.176.190
page_visits_10k 1 0 3334 3334 consumer-1_/10.139.176.190
page_visits_10k 2 0 3333 3333 consumer-1_/10.139.176.190

Hyades

Is there any API in JAVA to get it ? – Guadiana

consumer.assignment() – Belligerence

Answer 2

I know it does not directly answer the question, but still relevant.

If you want to see the IP address of the consumer that is currently connected to a partition use the following command

kafka-consumer-groups.sh --bootstrap-server my-kafka-1:9092 --group my-consumer-group  --describe --members

E.g.

# kafka-consumer-groups.sh --bootstrap-server my-kafka-1:9092 --group my-consumer-group  --members --describe --verbose | grep name-of-topic
my-consumer-group consumer-my-consumer-group-1-40adb273-fd3d-46a3-9cd4-9c8c4a4da486  /10.8.36.66     consumer-my-consumer-group-1  1               name-of-topic(1)
my-consumer-group consumer-my-consumer-group-1-47fc5618-7f07-4b3e-a2a9-cf3c26ee3536  /10.8.9.79      consumer-my-consumer-group-1  1               name-of-topic(2)
my-consumer-group consumer-my-consumer-group-1-005da938-7c59-44c3-8d4c-4371fa222d15  /10.8.36.22     consumer-pmy-consumer-group-1  1               name-of-topic(0)

Preoccupy