removing a kafka consumer group in zookeeper

Question

I'm using kafka_2.9.2-0.8.1.1 with zookeeper 3.4.6.

Is there a utility that can automatically remove a consumer group from zookeeper? Or can I just remove everything under /consumers/[group_id] in zookeeper? If the latter, is there anything else I'm missing & can this be done with a live system?

Update: As of kafka version 2.3.0, there is a new utility:

> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group

Related doc: http://kafka.apache.org/documentation/#basic_ops_consumer_lag

See below for more discussion

Clamant

Answer 1

Currently, as I know, the only way to remove a Kafka consumer group is manually deleting Zookeeper path /consumers/[group_id].

If you just want to delete a consumer group, there is nothing to worry about manually deleting the Zookeeper path, but if you do it for rewinding offsets, the below will be helpful.

First of all, you should stop all the consumers belongs to the consumer group before removing the Zookeeper path. If you don't, those consumers will not consume newly produced messages and will soon close connections to the Zookeeper cluster.

When you restart the consumers, if you want the consumers to start off from the beginning, give auto.offset.reset property to smallest (or earliest in new Kafka releases). The default value of the property is largest (or latest in new Kafka releases) which makes your restarting consumers read after the largest offset which in turn consuming only newly produced messages. For more information about the property, refer to Consumer Config in the Kafka documentation.

FYI, there is a question How can I rewind the offset in the consumer? in Kafka FAQ, but it gave me not much help.

Molar

Just cleaning up consumer groups that were misnamed or random console consumer groups.. Thanks! – Clamant

Related: What determines Kafka consumer offset? – Clamber

Note auto.offset.reset is ignored after there are offsets for a consumer group so they can pick up where they left off if shut down. This was confusing to me but confirmed from the developers. You can manually delete the group, or next run start a new group.id with the earliest or smallest flag and start over. Check out Burrow and has API for deleting groups and other maintenance and lag checking from team at LinkedIN. – Oosphere

If using Kafka's CLI, add the --from-beginning flag to your kafka-console-consumer ... call to replay the topic – Oosphere

Just to help anyone that searches for the commands on how to delete a consumer's path in zookeeper. First connect to zookeeper ./zookeeper-shell.sh localhost:2181, second check the node (that is optional) with get /consumers/<group-id>, and finally third delete the group with rmr /consumers/<group-id>. – Labyrinthodont

@Labyrinthodont consumer ids in zookeeper were generated automatically in my case. Is there a way to correlate the regular names that one gets via kafka-consumer-group.sh and the random ones via ls /consumers – Milesmilesian

Answer 2

As of v0.9.0, Kafka ships with a suite of tools in the /bin one of which is the kafka-consumer-groups.sh tool. This will delete a consumer group. ./kafka-consumer-groups.sh --zookeeper <zookeeper_url> --delete --group <group-name>

Polynomial

Thanks for the input, but this was not available in v0.8.1.1 (as stated in the question). It's available in the latest release (0.10.0.0) – Clamant

tar tvfz kafka_2.9.2-0.8.1.1.tgz | grep consumer | cut -c 49- kafka_2.9.2-0.8.1.1/bin/kafka-simple-consumer-shell.sh kafka_2.9.2-0.8.1.1/bin/kafka-console-consumer.sh kafka_2.9.2-0.8.1.1/bin/kafka-simple-consumer-perf-test.sh kafka_2.9.2-0.8.1.1/bin/kafka-consumer-perf-test.sh kafka_2.9.2-0.8.1.1/bin/windows/kafka-console-consumer.bat kafka_2.9.2-0.8.1.1/config/consumer.properties – Clamant

@foo-l I know it didn't exist for 0.8. Due to the kafka updates I thought this would help people currently searching for the issue. – Polynomial

If this doesn't work, delete from zk. rmr /consumers/<conusmer-group> – Moonlit

Answer 3

you can delete group from kafka by CLI

kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group group_name

Boys

the original question asked under a version of kafka which did not have the kafka-consumer-groups utility – Clamant

not possible, getting: Option '[delete]' is only valid with '[zookeeper]'. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires. – Milesmilesian

Answer 4

For new consumers (which use a kafka topic to manage offsets instead of zookeeper) you cannot delete the group information using kafka's built in tools.

Here is an example of trying to delete the group information for a new style consumer using the kafka-consumer-groups.sh script:

bin/kafka-consumer-groups.sh --bootstrap-server "kafka:9092" --delete --group "indexer" --topic "cleaned-logs"
Option '[delete]' is only valid with '[zookeeper]'. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

Here's the important part of that response:

Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

This is kind of annoying from a monitoring perspective (esp. when tracking offsets via something like burrow) because it means that if you change consumer group names in your code you'll keep seeing that old groups are behind on their offsets until those offsets expire.

Hypothetically you could write a tombstone to that topic manually (which is what happens during offset expiration) but I haven't found any tools that make this easy.

Madid