In a publish/subscribe model in microservices, how to receive/consume a message only once per service type

Question

We are designing for a microservices architecture model where service A publishes a message and services B, and C would like to receive/consume the message. However, for high availability multiple instances of services B and C are running at the same time. Now the question is how do we design such that only one service instance of B and one service instance of C receive the message and not all the other service instances.

As far as I know about RabbitMQ, it is not easy to achieve this behavior. I wonder if Kafka or any other messaging framework has a built-in support for this scenario, which I believe should be very common in a microservices architecture.

Donitadonjon

Answer 1

Kafka has a feature called Consumer Groups that does exactly what you describe.

Every identical instance of B can declare its group.id to be the same string (say "serviceB") and Kafka will ensure that each instance gets assigned a mutually exclusive set of topic partitions for all the topics it subscribes to.

Since all instances of C will have a different group.id (say "serviceC") then they will also get the same messages as the instances of B but they will be in an independent Consumer Group so messages go only to 1 of N instances of C, up to maximum number of instances which is the total number of topic partitions.

You can dynamically and independently scale up or down the number of instances of B and C. If any instance dies, the remaining instances will automatically rebalance their assigned topic partitions and take over processing of the messages for the instance that died.

Data never has to be stored more than once so there is still one single commit log or "source of truth" for all these service instances.

Clive

I'm new to Kafka so bear with me please. The documentations says each partition is assigned to only one consumer in a group, and there cannot be more consumer instances in a consumer group than partitions. Isn't this a limit in scalability of services B and C? Does it mean if I need to add more instances of my services I have to create a new partition as well? – Donitadonjon

It means you need to plan ahead if you want to avoid having to repartition a topic. It is not uncommon to start with 12 or 16 partitions for a topic so there is room to grow. If you only have 4 consumers to start they will just get 3-4 partitions each. If you run out of partitions you can add more. You just can't ever take them away. – Clive

Answer 2

Kafka has built-in support for this scenario.

You can create two Consumer Groups, one for B, and the other for C. Both Consumer Groups subscribe messages from A.

Any message published by A will be sent to both groups. However, only one member of each group can receive the message.

Salesgirl

Answer 3

This are the changes you need to perform to achieve the same with Rabbit MQ

• Create 2 seperate queue, one for each B and C service
• Change your logic to read message from queue such that only one instance will read the message from queue ,using blocking connection thing of rabbitmq.

this way, when multiple instance of B and C are running both will get the message and still be scalable.

Platitude

How do you publish a message to two or more queues at the same time? You send a message to an exchange with a key which is used to determine which queue is bound with that key. How do you define you binding keys? – Donitadonjon

Use broadcast exchange ,hence every message come to exchange will be broadcast to all the queue irrespective of key. – Platitude

What happens if one of the queues is unavailable/down and the other is up? Won't one queue get messages that the other will miss? If you had more services and therefore more queues wouldn't it be an administrative burden to ensure that they are all up and running at the same time, particularly since they might be on different nodes in the cluster and a single node failure would take only some of the queues out of service temporarily? – Clive

,system faliure is inevitable and solution for same is to have parallel nodes up and running for backup purposes. – Platitude

@VijayParmar, Can you please share some demo example or code to do this. – Lothario

Answer 4

You can also test this use case in kafka with the command line tools.

You create a producer with

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

Then, you can create two different Consumer Groups (cgB, cgC) with

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=cgB

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=cgC

As soon as you send a message to the topic, both groups (B,C) will receive the message but will save what message they processed independently.

Better explained here: Kafka quickstart

Publia