Redis, listening to pubsub events and turning them into a stream for more reliable consumption

Question

Using Redis, quite familiar. Now, I am facing a situation where the normal PUBSUB mechanism won't really be a great way to handle certain situations.

Take a hash where we say store a reference to another hash using say HSET.

HSET "davids.trips" "trip1" "Stockholm"
HSET "davids.trips" "trip2" "London"

HSET "david" "age"   "12"
HSET "david" "trips" "davids.trips"       # Reference to the previous hash

Now, the last one internally is known and handled as reference hash.

However, the problem we are facing now, is when expire now occurs on map "david" we would like the ability to invalidate the davids.trips as well so it does not linger along.

Now, we can pubsub to expire on HSET "david" say, but if our server is down then we won't be able to pick it up and it will be lost on us.

Doing this from Java and we run into the possibility of missing the expire pubsub message.

Instead, if we where to receive the message over a stream instead, the message would have to be consumed by at least one and we could ensure it is cleared.

Now, how can we do that?

Can we convert pubsub messages to stream like behaviour easily?

If so, how? I am using Jedis but I can send some code to potentially do this. Ideally, the conversion would happen within Redis server so that pubsub messages are never not transformed into Streams.

Btw on : https://redis.io/topics/pubsub we can find: "Please note that redis-cli will not accept any commands once in subscribed mode and can only quit the mode with Ctrl-C." So the problem is that it becomes blocking if we were to subscribe within Redis and then try to consume and turn them into streams.

There has to be a way to create a thread using lua and pubsub there?

Aerobiosis

We have the exact same problem. This can't be uncommon. And as yet, no solution. I can't even find a solution using another technology as yet... kafka? nope. And yet.. there is something like BullMQ that somehow figured out a way to do this... schedule a job (a record with a TTL?) and when the schedule time arrives, guarantees the job is delivered to a consumer (redis streams?).. but HOW they do it I have no idea. – Heikeheil

@Heikeheil do you believe it is possible to subscribe once on certain pubsub messages at redis? And upon recieval, it will publish on a stream instead? – Aerobiosis

Unfortunately.. no. Yang BoDong below mentioned Redis was looking into it... in 2020 or so. Unfortunately nothing ever came of it. It seems like such a very useful thing, no? But Redis 7.0 did NOT bring the feature in. – Heikeheil

I'm still very curious as to how Bull did it for node however. They.. figured something out, didn't they? Unless they are polling a list of tasks and then submitting them to a stream... I can't figure out any way to use the Pub Sub based on TTL to feed directly into a stream in a reliable way. – Heikeheil

Answer 1

Redis 7.0 will bring key space notifications with stream, may be this can solve your problem.

Unfolded

Unfortunately this answer didn't pan out. They have proposals, etc, but have yet to actually deliver a solution. – Heikeheil