Competing Consumer on Redis Pub/Sub supported?

Question

I have 2 services. Both of them need subscribe to the same channel.

The 2 services are load balanced. Each service runs on multiple servers.

So how can I be sure only 1 instance of each service consume the message of that channel.

Is this supported on Redis?

Thanks

Sewing

Answer 1

Pubsub doesn't work that way - the message goes to all connected subscribed clients. However, you could set it up so that the channel is a notification of an update to a list. That way all clients will get the message, but only one can take the item from the list with LPOP.

Languor

That's a great idea. Could you please explain more detail on "set it up so that the channel is a notification of an update to a list". thanks – Sewing

Just run two commands when you send a message - one RPUSH and one PUBLISH. When a client receives a message using SUBSCRIBE, have it call LPOP on the list. If LPOP doesn't return anything another worker has already processed the message so it can be ignored. – Languor

But then the publisher need to know the subscriber and break the observe pattern. e.g. Now my publisher knows there are 2 services consume the message. So I need send 2 RPUSH to 2 channel. Each channel map to a service. Not sure if I fully understand your idea? thanks – Sewing

If you don't want to change the publisher, you will need to flag the message status on the subscriber instead - something like SETNX service1receivedmessage1 1 combined with EXPIRE should work - SETNX will return 0 if another instance of the service already set the flag. – Languor

This is a really smart solution. So subscribers are talking to each other to make sure the service only execute once, although all servers received the message. thanks. If Redis can support competing consumer in pub/sub internally, it would be great. – Sewing

If you want to use a producer/consumer pattern over Redis you don't need to use pub/sub at all. Just have producer(s) LPUSH work into a list and have your consumer(s) BRPOP from the list (or multiple lists if you want consumers to service multiple different queues for different types of requests, for example). BRPOP will efficiently block until an event comes in, and each event will only go to and wake up one consumer. Think of it as a high level abstraction similar to the select() system call. (BRPOP also takes an optional timeout value, just like select() does). – Thug

I need pub/sub because my producer doesn't know how many consumers exists. It's just a notification. Anyone interested on it can subscribe it. But my situation is more complex. One consumer may start multiple instances on multiple servers. I need be sure only one instance of each consumer processing the notification. – Sewing

Answer 2

Another approach would be to use B*POP from your service instances. If you have lots of clients running B*POP against a list, whenever you LPUSH to it, one of those clients will get the data, but only one.

Skilling

Answer 3

You need use Redis Streams with XREADGROUP, it's a new feature of Redis.

https://redis.io/topics/streams-intro

Splice

Answer 4

I’ve implemented this provisionally by writing a value keyed by the request ID (a UUID). The requester writes the value with set, prior to the publish, and the consumer will attempt to delete this item, and only the consumer who successfully deleted the item will be considered the consumer that will process the request. I haven’t yet tested this at scale, so it may be the case that it doesn’t survive the scale I desire.

Border