Kafka: make consumer group Inactive

Question

I am running a data pipeline such that I read data from sql db into kafka topic through jdbc connect I sink this data in Elasticsearch using kafka sink connector for ES

I have a need to reset this pipeline. To that end I want to reset the consumer group that is listening on the jdbc connect topics. So I run the following command

kafka-consumer-groups --reset-offsets --to-earliest --all-topics --execute --group mygroup --bootstrap-server myserver:9092

But I get this error

Error: Assignments can only be reset if the group 'mygroup' is inactive, but the current state is Stable.

I stopped the connect to make the group inactive but that didnt work. So my question is, how do I make my group Inactive

Diapositive

Answer 1

--reset-offsets needs to check if a consumer is currently active before resetting the offsets. Stable indicates that the consumer is currently active and therefore offsets cannot be shifted back.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group mygroup --describe

should give any active consumer groups (if any). The output will help you identify which applications are currently active. Once the apps that keep consumers active are killed, you will be able to reset the offsets.

Alternatively, if you are using old consumers you can delete a consumer group(s) by running

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --group mygroup --delete

and finally reset the offsets.

Didymous

Answer 2

I just tried it with kafka 2.12 and when I got the same error I stopped the microservices which were holding this group (reading messages from the topics whithin this consumer group), and that solved the issue instantly. After that I could rescale the microservices and everything went on.

Bikol

Answer 3

As other answers explain how it works, it is because there is a consumer attached to the topic, so easiest thing to do is make the app inactive for some time.

You can also delete the consumer group but you might end up havin the problem described here

You can scale down your pods or services to zero or just shutting them down. Sometimes your pods will get restarted automatically but that gives you enough time to execute the command.

Rhythm