Redis Pub/Sub with Reliability

Question

I've been looking at using Redis Pub/Sub as a replacement to RabbitMQ.

From my understanding Redis's pub/sub holds a persistent connection to each of the subscribers, and if the connection is terminated, all future messages will be lost and dropped on the floor.

One possible solution is to use a list (and blocking wait) to store all the message and pub/sub as just a notification mechanism. I think this gets me most of the way there, but I still have some concerns about the failure cases.

1. what happens when a subscriber dies, and comes back online, how should it process all it's pending messages?
2. when a malformed message comes though the system, how do you handle those exceptions? DeadLetter Queue?
3. is there a standard practice to implementing a retry policy?

Habsburg

You can check the Redis patterns for a reliable queue at redis.io/commands/rpoplpush – Kiwanis

I too am having the same problem... I want to send location updates to clients.... and once they disconnect, i dunno how to synchronize the data between client and server... Did you solve the issue? If yes, how?? – Grosberg

Answer 1

When a subscriber (consumer) dies, your list will continue to grow until the client returns. Your producer could trim the list (from either side) once it reaches a specific limit, but that is something you would need to handle at the application level. If you include a timestamp within each message, your consumer can then act on the age of a message, assuming you have application logic you want to enforce on message age.

I'm not sure how a malformed message would enter the system, as the connection to Redis is usually TCP with the its integrity assurances. But if this happens, perhaps due to a bug in message encoding at the producer layer, you could provide a general mechanism for handling errors by keeping a queue-per-producer that received consumer's exception messages.

Retry policies will depend greatly on your application needs. If you need 100% assurance that a message has been received and processed, then you should consider using Redis transactions (MULTI/EXEC) to wrap the work done by a consumer, so you can ensure that a client doesn't remove a message unless it has completed its work. If you need explicit acknowlegement, then you could use an explicit ACK message on a queue dedicated to the producer process(es).

Without knowing more about your application needs, it's hard to know how to choose wisely. Generally, if your messages require full ACID protection, then you probably also need to use redis transactions. If your messages are only meaningful when they are timely, then transactions may not be needed. It sounds as though you can't tolerate dropped messages, so your approach of using a list is good. If you need to implement a priority queue for your messages, you can use the sorted set (the Z-commands) to store your messages, using their priority as the score value, along with a polling consumer.

Ape

Answer 2

If you want a pub/sub system where subscribers won't lose messages when they die, consider using Redis Streams instead of Redis Pub/sub.

Redis Streams have their own architecture and pros/cons to Redis Pub/sub. With Redis Streams, a subscriber can issue the command:

the last message I received was X, now give me the next message; if there is no new message, then wait for one to arrive.

Antirez's article linked above is a good intro to Redis streams with more info.

Nader

Answer 3

What I did is use a sorted set using the timestamp as the score and the key to the data as the member value. I use the score from the last item to retrieve the next few ones and then get the keys. Once the work is done I wrap both the zrem and the del in a MULTI/EXEC transaction.

Essentially what Edward said, but with the twist of storing the keys in the sorted set, as my messages can be pretty big.

Hope this helps!

Succor