Kafka: How to Display Offsets

Question

I'm super new to Kafka. I've installed kafka and zookeeper using homebrew on my mac, and I'm playing around with the quickstart guide.

I've been able to push messages onto Kafka using the following command and STDIN

kafka-console-producer --broker-list localhost:9092 --topic test

and I can read things off using

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

What's not clear to me is how I use offsets. It's my understanding that each message added to a topic will have a numerical, incremental offset value. However, if I try to do something like this

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset 1

I get a non-zero status code and shows me no messages (other than the usual help/usage information)

I also can't get the latest or earliest keywords to work

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset earliest
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset latest

Both of the above also return non-zero status codes.

Do I fundamentally misunderstand offsets? If not, is there a way to list all messages with their offsets? Finally -- what's the simpliest example of the --offset flag for the kafka-console-consumer?

Whallon

Hey, after looking at the source code for console consumer it appears there's no option to set offset to some value as you desire (you can do that with the actual consumer, by using the seek api call). latest is the default (that is, if you start the console consumer without --from-beginning it will only receive "new" data). --from-beginning is the same as "earliest". You may use the group option which allows you to reuse the same group id, guaranteeing that consecutive runs of the consumer won't receive duplicate messages. – Cecilececiley

Thank you @RoyShahaf that's super useful (especially the group info) -- random follow up question if you have a moment: "looking at the source code for console consumer" -- where did you look at the source code for this? – Whallon

I'm glad I could help. Regarding your followup question: I googled kafka-console-consumer.sh which lead me to some github with some file named kafka-console-consumer.sh which seemed to be running a ConsoleConsumer so I googled that and found something that looked relevant :) – Cecilececiley

Answer 1

If you looked at the output after you gave the offset value, it would have said that you needed to specify a partition (at the top of the help section)

Topics are subdivided into partitions, and an offset of 1 could only exist on one partition out of potentially hundreds, so you must specify it

Regarding displaying the offsets, lookup the GetOffsetShell command syntax

Sukey

Ah ha, thank you, super useful! I was distracted by the "let's output all the help" behavior whenever there's an error. I'll endeavor to RTFE in the future :) – Whallon

Answer 2

A nice command line tool that can display the offset with each message is kafkacat.

kafkacat -b localhost:9092 -C -t test -f 'Topic %t [%p] at offset %o: key %k: %s\n'

It will print out something like

Topic test [5] at offset 111: key "0171bf8102007900e33": {"Message": "1"} 
Topic test [2] at offset 123: key "070021b0f001f614c1b": {"Message": "2"}

Willawillabella

Answer 3

Since GetOffsetShell only works with PLAINTEXT, it can be inconvenient for many.
Good news is that additional properties, including print.offset seems to have made their way into 2.7 based on the communication in 9099 PR.
It must be possible to use print.offset=true now!

Klug

You can get the full list of the properties from cwiki.apache.org/confluence/display/KAFKA/… – Acrodrome

Answer 4

Partition is required when using offset

.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --partition 0 --offset 0 --property print.key=true --property print.timestamp=true

Elspet

Answer 5

You have to give partition value as well with offset in above command like this

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --partition 0 --offset 1

Escapism

Answer 6

here is the answer, just use '--property print.offset=true':

bin/kafka-console-consumer.sh --bootstrap-server XXXXXX --topic T-TEST --partition 0 --offset 0 --property print.offset=true 

Buiron