How Kafka guarantee the messages order while we increase the partitions in runtime?

Question

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

If so, what if all messages are provided with keys, then none of them will be mapped to new partition? This doesn't make sense to me.

If not, then how Kafka guarantee the order to messages with the same key in order?

Chiapas

Answer 1

I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

No, the new messages will be partitioned based on the new number of partitions. Old messages will not get re-partitioned.

If not, then how Kafka guarantee the order to messages with the same key in order?

There are no guarantees when changing the number of partitions. When increasing the number of partitions at runtime the ordering can, or better, will change.

Changing a partition of a topic that already includes data is therefore not recommended if you are relying on ordering of your data. What you could do is

• to copy the existing data into a new topic with the new number of partitions before having your producer send data to it.
• apply a custom partitioner in your producer that sends messages based on a key into a fixed partition and only have "new keys" send to the new partitions.

Remember, it is not possible to decrease the number of partitions.

Connally