joining Redis streams

Question

I am very intrigued by Redis streams. (Looks like the potential to build little systems powered by append-logs, like Kafka, but without all the overhead of Kafka.)

It looks straightforward to XADD to a log/stream and to consume an entry from a log/stream. But what about if you want to join across two streams?

Kafka Streams, Flink, Spark, etc. provide means for doing this. Is there an equivalent in the Redis universe?

If not, I guess I'll just need to implement my own thing that consumes from two streams, does its own join logic from the messages, and publishes back out to a new stream. If others have experience doing this with Redis Streams, please do share your pointers or warnings.

Homomorphism

I'd use it. Lets us know here. – Bonnice

It seems for Flink there is a Redis sink. This may be a good starting point to make a Redis source, and after that you will have the full capabilities like joining automatically available through Flink. – Deserving

Answer 1

If I am correct, you are looking for a way to join two Redis streams.

It seems there is a connector available for Spark, that allows you to consume the streams https://github.com/RedisLabs/spark-redis/blob/master/doc/streaming.md

From here Spark logic for the join should be easy to use.

Deserving

Answer 2

I have attended the Redis University Stream course and they does not show any built in method to join messages from two streams.

I guess one work-around would be to handle the IDs in a proper way. Let's make an example:

You produce messages to stream A with even IDs.
You prodce messages to stream B with odd IDs.
You consume from stream A and stream B and produce to a new stream C being careful on respecting the order because it is not allowed to produce an ID equal or smaller than the larger already present in the stream.

In this whay you achieve the join.
Looking forward to better answer not using external libraries

Blossomblot