Kafka - Consumer group creation with specific offset?

Question

After creating a topic in Kafka, you can create an arbitrary number of consumer groups just by trying to use those groups to read from the topic.

I would like to create an extra consumer group for monitoring the message content of the real consumer groups - one used to peek at their messages. So, GUI would let you click "peek" on any consumer group, and the "peeker" group would have its offset updated to the offset of the group being monitored, then it would show you the message from that offset.

I'm confused though, because you can't explicitly create a consumer group the first time; you seem to have to read a message to get the offset node created in zookeeper.

My Question

Is there a way to explicitly create a consumer group pointed at a specific offset, or is it okay to manually create the zookeeper node for a consumer group that has't been used yet so that it is initialized to the correct offset value? Or will this auto-creation mess up the consumer group allocation process?

Tussis

Answer 1

You could have a look at the Kafka Web Console project which already does something similar to what you describe.

If you want to do this yourself, you'd need to use the simple consumer API and manually handle offsets for a new consumer group (stored in Zookeeper or elsewhere). You could get the current offset from the existing consumer group and then read messages using the same offset for your peek group. As long as the group ids are different, they should not interfere with each other or mess anything up.

Ziwot

Answer 2

For readers, Kafka Web Console is no longer supported. Please consider Kafka Manager instead.

Murphey

Answer 3

As noted above Kafka Manager has a really nice interface and would be well worth your time to setup. But if you want the CLI version, as I needed, the below should work:

groupId="legitGroupId"
kafka="localhost:9092"
declare -a topics=(
    "topic1" 
    "topic2"
    )

# Create a single consumer of all the topics which starts starts at each topics latest offset
# Use --dry-run instead of --execute to see how the end results will look
for topic in "${topics[@]}"; do
    echo "Adding consumer to $topic"
    kafka-consumer-groups --bootstrap-server $kafka --topic $topic --group $groupId --execute --reset-offsets --to-latest
    echo ""
done

Stenograph