How to send message to a particular partition in Kafka?

Question

I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this,

kafka-console-producer.bat --broker-list localhost:9092 --topic sample  --property parse.key=true --property key.separator=,

Send messages as,

key1,another-message

But I am just confused on whether key1 represents partition number.

Using the console consumer I viewed the messages,

kafka-console-consumer.bat --zookeeper localhost:2181 --topic sample

I want to view the messages according to the partitions. Is this the right way to view the messages on the console consumer? Can anyone please provide a clear understanding on this?

Procurance

Note that if you are not using the console producer but a Kafka producer client (like the official Java producer client) you can actually directly specify the target partition that a message should be sent to. The console producer is nice for playing around but typically not used in production. – Whitehouse

Possible duplicate of How to produce messages to selected partition using kafka-console-producer? – Arsis

Answer 1

You can specify partition number directly in the ProducerRecord, but not with kafka-console-producer.

The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

If you need a custom algorithm to map the messages to partitions, you need to implement org.apache.kafka.clients.producer.Partitioner interface. The name of you class must be set as a partitioner.class property of the producer.

Helli

Custom partitioning can be done with programming, one of the examples can be found here. You cannot specify it with a kafka-console-producer – Helli

But this example is a Java code, do you have the same in Scala? – Procurance

Is there any way so that we can specify partition number while sending? – Procurance

Yes, create a ProducerRecord with ProducerRecord(java.lang.String topic, java.lang.Integer partition, K key, V value) Creates a record to be sent to a specified topic and partition – Helli

So how can we identify the partition number? can you give an example? – Procurance

What is the difference in using partition number and key. I am getting confused. – Procurance

Producer uses the hash of the key to distribute the message to partitions. For example, you can have two partitions, the integer key and have algorithm to distribute the even keys to the first partition and the odd keys to the second partition. As for partition number, when you send the message to Kafka you use ProducerRecord object. With this object you can specify partition number explicitly. – Helli

Ok I got it. Thank you – Procurance

Please could you advise what happens when the key (or hash) is the same for more than one messages? That is, let's say I have 3 partitions, and I try to write two messages with the exact same key to a topic? Does it matter if the message body is the same or different? Thanks! – Brahmin

Keys in Kafka are not unique. So, if you have 3 messages with the same key, all the 3 messages will be written into Kafka. It does not matter if they have the same or different body. If you have partitioning algorithm based on key, all the 3 messages will go to the same partition. – Helli