How to add partition to Kafka topic and keep same-key message in same partition？

Question

It is common to require ordering in same partition of given Kafka topic. That is, messages with same key should go to same partition. Now, if I want to add new partition in a running topic, how to make it and kept the consistency?

To my understanding, the default partitioning strategy is to mod on num-of-partition . When the num-of-partition changes (e.g. 4 to 5), some messages might fall into different partition from previous messages with same key.

I can image to have consistent hashing implemented to customize the partitioning behavior, but it might be to intrusive.

Or, just stop all producers until all messages are consumed up; then deploy new partition and restart all producers.

Any better ideas?

Gook

Answer 1

As you said, when you increase the number of partitions in a topic you will definitely loose the ordering of messages with the same key.

If you try to implement a customized partitioner to have a consistent assignment of a key to a partition, you wouldn't really use the new partition(s).

I would create a new topic with the desired amount of partitions and let the producer write into that new topic. As soon as the consumers of the old topic have processed all messages (i.e. consumer lag = 0) you could let the consumers read from the new topic.

Farmland

But isnt Kafka uses consistent hashing to tackle this issue? toptal.com/big-data/consistent-hashing and here medium.com/omarelgabrys-blog/… – Kruse