Can a message loss occur in Kafka even if producer gets acknowledgement for it?

Question

Kafka doc says:

• Kafka relies heavily on the filesystem for storing and caching messages.
• A modern operating system provides read-ahead and write-behind techniques that prefetch data in large block multiples and group smaller logical writes into large physical writes.
• Modern operating systems have become increasingly aggressive in their use of main memory for disk caching. A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and writes will go through this unified cache
• ...rather than maintain as much as possible in-memory and flush it all out to the filesystem in a panic when we run out of space, we invert that. All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel's pagecache.”

Further this article says:

(3) a message is ‘committed’ when all in sync replicas have applied it to their log, and (4) any committed message will not be lost, as long as at least one in sync replica is alive.

So even if I configure producer with acks=all (which causes producer to receive acknowledgement after all brokers commit the message) and producer receives acknowledgement for certain message, does that mean their is still a possibility that the message can get lost, especially if all brokers goes down and the OS never flushes the committed message cache to disk?

Hunnish

Answer 1

With acks=all and if the replication factor of the topic is > 1, it's still possible to lose acknowledged messages but pretty unlikely.

For example, if you have 3 replicas (and all are in-sync), with acks=all, you would need to lose all 3 brokers at the same time before any of them had the time to do the actual write to disk. With acks=all, the aknowledgement is sent once all in-sync replicas have received the message, you can ensure this number stays high with min.insync.replicas=2 for example.

You can reduce the possibility of this scenario even further if you use the rack awareness feature (and obviously brokers are physically in different racks or even better data centers).

To summarize, using all these options, you can reduce the likeliness of losing data enough so that it's unlikely to ever happen.

Walden

(1) Does this makes kafka unfit for use case scenarios where message loss is not tolerable? Or people still use kafka successfully even for such scenarios? (2) Does any other queueing system (RabbitMQ, ActiveMQ etc) ensure persistence of message once its acknowledge and hence no message loss (at least not due to queue server temporary down.) – Hunnish

While it's "possible", if you have at least 3 replicas, on brokers in different datacenters, it's extremely unlikely. That does not make Kafka unfit for scenarios where data loss is not tolerable. – Walden

Even if the messages were written to disk, in theory the disk of all your brokers could break down at the same time. But here again, it's extremely unlikely to happen. – Walden

Answer 2

In addition to these, there are 2 other important configuration parameters:

• linger.ms=0 (batching on production side)
• auto.commit=false

I would avise you read this doc: https://developer20.com/when-you-can-nose-messages-in-kafka/

When the data loss can happen and how to prevent it.

Publisher: Acknowledgment When a message is sent to the publisher, the publisher waits for an acknowledgment (ACK) from the broker. There are three configuration options which can be used:

acks = all - the broker will return the ACK only when all replicas will confirm that they saved the message. acks = 1 - the ACK will be returned when the leader replica will save the message but won’t wait for replicas to do the same acks = 0 - the producer won’t wait for the confirmation from the leader replica As the last option is obvious (fire and forget), the second one may lead to less explicit data loss. There is a scenario when the producer will receive confirmation that the message was saved, but just after the ACK the leader replica crashes and doesn’t start. Because other replicas don’t have the message when a new replica leader is elected the message is gone forever.

By default, the acks is set to 1 in JVM and all in golang. As you can see, there are differences in various implementations, so it’s a better idea to set this value explicitly.

Publisher: The buffer For performance reasons (to reduce the network usage) a buffering can be enabled. Messages aren’t sent when the publishing method is called but when the buffer reached its maximum capacity or in a given interval. Those behaviors are controlled by batch.size (in bytes) and linger.ms parameters. If one of those limitations is reached, the messages are sent at once. What’s important to stress, the client will receive information that the message is already sent, but that’s not true. If the app crashes before flushing the buffer, the data are irreversibly lost.

Please remember that those parameters may be different depending on the implementation. In JVM the batch.size is a number of bytes in the buffer (16384 bytes by default), but in kafka-go the parameter describes the number of messages in the buffer (100 by default). What’s more, the JVM users have the linger.ms set to 0 by default but kafka-go users have set it 1 second.

In the JVM implementation, when batching is disabled (linger.ms=0) messages can still be sent together. It happens under heavy load - the messages that arrive close together in time will be batched anyway.

Subscriber: Offsets During consuming messages, the consumer (subscriber) sends his current offset to the broker. This is the place where data loss can happen. There are, at least, two plots when it may happen.

The first scenario is consuming in parallel. Imagine a situation where 2 messages come to a consumer: A and B. All the messages are processed in parallel. Processing the messages, B was successful, and the offset was committed. However, handling the message, A produced an error. Because the message B has a larger offset, Kafka will save the latest offset and the message A never comes back to the consumer.

Broker: Committed doesn’t mean saved on the disk Kafka, on Linux system, saves messages to a filesystem cache but doesn’t wait the message get persisted on the hard drive. It means that if you have only one replica or acks = 1 it is possible that the broker will go down and the message will be lost even if the broker returned the ACK.

Broker: Saved on the hard drive doesn’t mean it won’t disappear Not all the data that exists on the leader of the partition is available for clients to read. It happens when not all in-sync replicas received the message. For example, when a follower broker is behind the leader but it is still considered as in-sync (the lag time is configured by replica.lag.time.max.ms parameter, 500 by default) and then the leader crashes. A new leader is elected, but it didn’t receive the message. The message is gone. This situation is the reason why consumers are not allowed to receive unsafe data.

Cohune

Answer 3

What Mickael is correct, but there's a possibility of Kafka loosing data even when there are multiple in-sync replicas.

The problem is that Kafka relies on replication to ensure that no data is lost, rather than ensuring data is stored in permanently by doing an fsync. The problem happens when a node receives data, replicates it, stores it (without fsync), sends an ACK and the node dies. In this case the replica has all the data, but the storage might not have saved all of the messages. When the node is brought up again and resumes being the master for the partitions, it will think that everything is fine, but will have lost some data. If a consumer then replays all the messages, that message (or batch) won't be returned.

There's a more detailed write up from Red Panda on the blog Why fsync(): Losing unsynced data on a single node leads to global data loss. Bear in mind that Red Panda is a competitor of Kafka.

Concupiscent