Kafka Consumer with enable.auto.commit = false still committing the offsets

Question

Kafka consumer has processed the messages 1, 2, 3, 4 and the enable.auto.commit is set to false.

But on restarting the consumer, it is not reprocessing the above messages again, from CLI I could see that the offset has been incremented and there is no lag (hence it is committing).

Can you please help on this, to understand how the consumer is still committing the offsets though the property enable.auto.commit is set to false.

Below are the consumer properties

allow.auto.create.topics = true
auto.commit.interval.ms = 0
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
enable.auto.commit = false
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = EmployeeConsumer

Currently using the spring-kafka-2.5.0.RELEASE.jar as the dependency

Rudnick

Answer 1

You need to show your Spring configuration.

enable.auto.commit=false tells the kafka-clients not to commit offsets, but Spring will commit offsets by default.

Set the listener container ackMode property to AckMode.MANUAL to disable the container commits.

Furthermore:

auto.offset.reset = latest means that a consumer that has never committed an offset will start consuming from the current end of the topic/partition so it won't get existing records.

Ambrogino

@Bean public ConcurrentKafkaListenerContainerFactory<Long, Employee> kafkaListenerContainerFactory() { ConcurrentKafkaListenerContainerFactory<Long, Employee> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.getContainerProperties().setAckMode(AckMode.MANUAL); factory.setConsumerFactory(consumerFactory()); return factory; } It worked! – Rudnick

@Gary Russell So if both are set to true then do they override each other at time of storing offsets? – Kaolack

I am not sure what you mean. Only one of the described properties is boolean. If auto.offset.reset is true, Spring is hands off since you want the kafka-clients to be in control. This is not recommended. Spring uses a more determinative algorithm. – Ambrogino

Answer 2

The property:

auto.offset.reset = latest

This will make sure you will always read the messages which were being added on or after your consumer starts. So enabling or disabling auto commits won't have any impact on data your consumer will process.

if you want to process all the data you should set it to

auto.offset.reset = earliest

you can refer this doc : auto.offset.reset

Vladimir