Python-redis: get binary data after a client was set up with decode_responses=True

Question

So, following the excellent suggestion in both this answer and that answer, I decided to replace a whole bunch of encode/decode to/from UTF-8 all over the place by a single:

rdb = redis.StrictRedis(..., encoding='utf-8', decode_responses=True)

But then, as others have pointed out in comments to the answers above, that connection is then unable to "handle binary data". (Small point: I slightly disagree with that: "decode_responses" is well-named: the responses are unconditionally converted from binary to string, but arbitrary binary data can still be stored, just not retrieved).

So, absent of having a way to briefly override the decode_responses setting for a single query, I was wondering if there was a way to derive a new client from an existing one, with largely the same parameters? That way, I could make a new client with decode_responses=False just to retrieve data I know to be binary.

Nino

Answer 1

Here is what I came up with. Not sure how it would handle complex connections and what else it may break. Just don't run your self-driving car with that...

def new_client(client, **kwargs):
    """return a new Redis client based on an existing one,
    with some kwargs modified.
    """
    kwargs = {**client.connection_pool.connection_kwargs, **kwargs}
    return redis.StrictRedis(**kwargs)

With this, now we can do, e.g.:

client.set(name, pickle.dumps(stuff))

...

# later
with new_client(client, decode_responses=False) as binclient:
    data = binclient.get(name)
stuff = pickle.loads(data)

Nino

This is a really bad idea as introduces a gaping hole unless you can absolutely guarantee nobody can ever under any circumstances mess with what you have in redis. Details here: https://mcmap.net/q/197351/-pickle-or-json-duplicate/… and here: docs.python.org/3/library/pickle.html Please consider removing your answer. – Ronaronal

@ostergaard: appreciate your input; I'll take a look as soon as I have a second and will update accordingly. Please also feel free to add another answer if you have a better solution. – Nino

I don't really have a better answer - you really just need to swap your pickle.dumps and pickle.loads for json.blah as described in multiple other answers. – Ronaronal

Answer 2

My approach is to open two connections: one for decoding/encoding and one for raw (not sure if this results in a performance or memory problem, but it works for me):

class RedisPool:
    def __init__(self):
        self.REDIS_HOST: str = HOSTS.REDISDB or '127.0.0.1'
        self.REDIS_PORT: int = PORTS.REDISDB or 6379

        self.pool = redis.ConnectionPool(
                        host=self.REDIS_HOST,
                        port=self.REDIS_PORT,
                        db=0,
                        protocol=3,
                        decode_responses=True)



class RedisPoolRaw:  # raw binary
    def __init__(self):
        self.REDIS_HOST: str = HOSTS.REDISDB or '127.0.0.1'
        self.REDIS_PORT: int = PORTS.REDISDB or 6379

        self.pool = redis.ConnectionPool(
                        host=self.REDIS_HOST,
                        port=self.REDIS_PORT,
                        db=0,
                        protocol=3,
                        decode_responses=False)

Floranceflore