How change Kafka committed consumer offset with required offset

Question

I have Kafka Stream application. My application is processing the events successfully.

How to change Kafka committed consumer offset with required offset to reprocess/skip the events. I tried How to change start offset for topic?. But I got 'Node does not exist:' error. Please help me.

Cabinda

Answer 1

The question/answer you are referring to is based on an older Kafka version. Since Kafka 0.9, offsets are not committed to ZooKeeper but store in a special Kafka topic called the offset topic (topic name is __consumer_offsets).

Since Kafka 1.0, the command line tool bin/kafka-consumer-groups.sh has a new feature that allow to set offsets. Check out the original KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling

The tool also works for Kafka 0.11 (and maybe even older Kafka versions).

An alternative is to write your own tool, that uses an single KafkaConsumer with the corresponding group.id, subscribe to the topic(s) you want to modify offsets for, seek() and commit() offsets. (Note, you should disable auto commit for this consumer.)

Winifredwinikka

I tried to reset the offset as per document. But I got unrecognized option exception. My Kafka verstion is 0.10.2.0 Command is: /home/centos/kafka_2.12-0.10.2.0/bin/kafka-consumer-groups.sh --bootstrap-server 1.1.1.1:9092,1.1.1.2:9092 --reset-offsets --group sample-app-0.0.1 --topic abc --to-offset 76756952 Exception: Exception in thread "main" joptsimple.UnrecognizedOptionException: reset-offsets is not a recognized option. Is this option not available in 0.10.2.0. – Cabinda

Because you call kafka_2.12-0.10.2.0/bin/kafka-consumer-groups.s‌​h you are using the old tool. You need to download Kafka 1.0 and use the tool from there. – Winifredwinikka