Kafka consumer not consuming from beginning

Question

I have Kafka setup on my local machine and have started the zookeeper and a single broker server.

Now i have a single topic with following description:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic edu-topic --describe
Topic:edu-topic PartitionCount:3    ReplicationFactor:1 Configs:
    Topic: edu-topic    Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: edu-topic    Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: edu-topic    Partition: 2    Leader: 0   Replicas: 0 Isr: 0

I have a producer which have produced some message before the consumer was started as follows:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic edu-topic
>book 
>pen 
>pencil
>marker
>

and when i started the consumer with --from-beginning option, it does not shows all the messages produced by the producer:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning

However, it is showing the newly added messages.

What's wrong i am doing here? Any help?

Dilorenzo

Answer 1

--from-beginning: If the consumer does not already have an established offset to consume from, start with the earliest message present in the log rather than the latest message.

Kafka consumer uses --from-beginning very first time if you retry which I suspect you did, it will start from where it left. You can consume the message again with any of the below options

1. reset consumer group offset using below

kafka-streams-application-reset.sh --application-id edu-service --input-topics edu-topic --bootstrap-servers localhost:9092 --zookeeper 127.0.0.1:2181

then retry again from the beginning

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning

2. Use new consumer id which will start consuming from staring points

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group new-edu-service --from-beginning

3. You can also use an offset instead to consume the next N messages from a partition

kafka-console-consumer.sh --bootstrap-server localhost:9092 --offset 0 --partition 0 --topic edu-topic

--offset <String: consume offset> : The offset id to consume from (a non- negative number), or 'earliest' which means from beginning, or 'latest' which means from end (default: latest)
--partition <Integer: partition> : The partition to consume from Consumption starts from the end of the partition unless '--offset' is specified.

Stipulation

Another option is resetting offset to earliest with kafka-consumer-groups, kafka-consumer-groups --bootstrap-server localhost:9092 --group edu-service --topic edu-topic --to-earliest --reset-offsets --execute. This becomes available in v0.9.0. – Nett

Answer 2

Because you are using the old consumer group. --from-beginning only works for the new consumer group which its group name has not been recorded on the Kafka cluster yet.

To re-consume again from the start, either you can:

• Start a new consumer group (change the group name) with the flag --from-beginning
• Reset the offsets of this consumer group. I haven't tried yet but you can test it here

Casebook

Answer 3

The flag

--from-begining

will affect the behavior of your GroupConsumer the first time it is started/created , or the stored (last commited consuming) offset is expired (or maybe when you try to reset the stored offset).

Otherwise the GroupConsumer will just continue at the stored (last commited) offset.

Please consider get more message from manual.

Constructivism

Answer 4

Just add --from-beginning

But do know that messages from the beginning would not be in order

if you have used multiple partitions for the same topic. Order is only Guaranteed at the partition level. (for the same partition)

Kale