Removing one message from a topic in Kafka

Question

Can I delete only ONE message from a topic if I know the topic, the offset and the partition?

And if not is there any alternative?

Dammar

Answer 1

It is not possible to remove a single message from a Kafka topic, even though you know its partition and offset.

Keep in mind, that Kafka is not a key/value store but a topic is rather an append-only(!) log that represents a stream of data.

If you are looking for alternatives to remove a single message you may

1. Have your consumer clients ignore that message

2. Enable log compaction and send a tombstone message

3. Write a simple job (KafkaStreams) to consume the data, filter out that one message and produce all messages to a new topic.

Sickness

Considering our event stream context, there's a fourth option that aligns more with the event-sourcing paradigm. Instead of attempting to tombstone an erroneous message, you could consider appending a new <ConsumerNameHere>Failed event to the end of the topic. This approach involves shifting the pointer within your consumer. The <ConsumerNameHere>Failed event could encompass a reference to the original unprocessed event along with a record of the attempted actions. This technique allows the consumer's pointer to progress while retaining the means to retry the problematic event :) – Boyar

P.S. the above solution would only be appropriate if you wanted to advance the consumer pointer and retry at the end of the current stream, but hopefully that's clear from context. – Boyar