Does the redis-py module work with Redis in Cluster mode?

Question

I'm trying to use redis-py with redis in cluster mode, but I can't get it to work. I see that redis-py-cluster works, but I have a preference for redis-py as I have been using it and it's a recommended client.

Yale

Answer 1

redis-py does not support cluster mode. Clustering has totally different architecture to serve the purpose of horizontal scalability. HA (High Availability) was not a priority in its design. Therefore you can not use one client for the other.

redis-py-cluster seem to have ongoing development / support, and it's based on redis.py. The client page you linked was not for redis cluster. "redis-py-cluster" is mentioned on redis cluster page (look for "Playing with the cluster"): https://redis.io/topics/cluster-tutorial

Asides from clustering, Redis has sentinel supported setup to provide HA, which redis-py does support.

Flieger

redis-py 4.1.0 added support for Redis cluster. See github.com/redis/redis-py/releases/tag/v4.1.0rc1 Seems like there have been several fixes, so use the most recent version of redis-py? – Syndetic

Answer 2

According to redis-py documentation:

redis-py is now supports cluster mode and provides a client for Redis Cluster.

Notice that the redis-py added this feature in version 4.1.0 which has no stable release right now. if you want to have it installed, you should use the command below:

pip install redis==4.1.0-rc1

Maybe when you are reading this answer, it is stable! So just install without the -rc1 postfix.

You can connect to your redis-cluster like the following:

>>> from redis.cluster import RedisCluster as Redis
>>> rc = Redis(host='localhost', port=6379)
>>> print(rc.get_nodes())
[[host=127.0.0.1,port=6379,name=127.0.0.1:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6379,db=0>>>], [host=127.0.0.1,port=6378,name=127.0.0.1:6378,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6378,db=0>>>], [host=127.0.0.1,port=6377,name=127.0.0.1:6377,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6377,db=0>>>]]

Thigpen

Answer 3

You can use redis-py in redis cluster, but as different keys are partitioned to different nodes, you need to compute (by crc16/crc32 hash function) which cluster handles which keys.

In order to fully utilize "cluster mode", where you don't care about keys' location, you need to implement "client side partitioning" and "query routing" which redis-py-cluster provides. (https://redis.io/topics/partitioning)

One major disadvantage of redis-py-cluster is that it does not provide a solution for the atomic operation in "pipeline + transaction"

Wunder

There are different methods and implementations to solve the pipeline + transaction problem. But the main problem is that it is impossible to make a solution right now in a clustered redis environment where the entire pipeline is atomic or transactional. You can only get as far as having transaction/atomic per node inside a pipeline but never the entire pipeline. More changes would have ot be needed inside redis to support this feature fully. – Deviltry