Apache Kafka: Can we restrict message to be read by only 1 consumer?

Question

I am new to Kafka and the scenario that i am trying to achieve using Apache Kafka is:

I have a 3 producers who are sending instructions to a particular topic in Kafka. I have around 35 consumers who picks up one instruction at a time and performs the task mentioned in the instruction set. Once completed it will post the updates to an another topic in Kafka.

Requirements.

1. An instruction read by 1 consumer should not be given to another for processing
2. Message Priority * - Not a mandatory requirement
3. Only 1 instruction should be picked up at a time for processing.

I know that this can be achieved using JMS. But my organisation already has a Kafka setup ready as part of Cloud-era HDFS setup. So the using Kafka to achieve this is most preferred.

I have googled around to find a solution for this. But i was not able to find a proper answer to my 1st and most important requirement.

Any suggestions? And thanks in advance.!

Mechanics

Have a look here: stackoverflow.com/documentation/apache-kafka/5449/… – Torrie

Furthermore, you can configure each consumer to receive a single message per call via parameter "max.poll.records=1". See kafka.apache.org/documentation#newconsumerconfigs – Torrie

Answer 1

All you have to do is to put the consumers in the same consumer group by assigning the same group.id to them. Only one instance of a consumer in the same group will get a message.

Refer to documentation at https://kafka.apache.org/documentation#intro_consumers

Quantity

Also see parameter "max.poll.records" kafka.apache.org/documentation#newconsumerconfigs – Torrie

For python kafka-python package, KafkaConsumer class supports 'group_id' parameter to assign consumer group – Elan