Handle redis connection reset

Question

We are facing [Errno 54] Connection reset by peer at very random in our application and looks like it is being triggered by redis server than client. Python's redis client have backoff strategy implementation but it's unable to handle this scenario.

There are github issues on official repo as well and many people commented recently confirming this problem.

• Random ConnectionErrors (104, Connection reset by peer) #1186
• redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer') #1439

Step to reproduce

$ ipython
# in python client
import redis
from redis.retry import Retry
from redis.exceptions import (TimeoutError, ConnectionError)
from redis.backoff import ExponentialBackoff

# connect client with exponential backoff retry
client = redis.StrictRedis(retry=Retry(ExponentialBackoff(cap=10, base=1), 25), retry_on_error=[ConnectionError, TimeoutError, ConnectionResetError], health_check_interval=1)

client.keys()
# print all keys

Now reset connection directly from redis server

$ redis-cli
RESET

Wait for 120 second or more and run client again

# in client
client.keys()
---------------------------------------------------------------------------
ConnectionResetError                      Traceback (most recent call last)
<ipython-input-91-011ce9f936fc> in <module>
----> 1 client.keys("rq*")

~/path-to-python-env/env/lib/python3.8/site-packages/redis/commands/core.py in keys(self, pattern, **kwargs)
   1386         For more information check https://redis.io/commands/keys
   1387         """
-> 1388         return self.execute_command("KEYS", pattern, **kwargs)
   1389 
   1390     def lmove(self, first_list, second_list, src="LEFT", dest="RIGHT"):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/client.py in execute_command(self, *args, **options)
   1168         pool = self.connection_pool
   1169         command_name = args[0]
-> 1170         conn = self.connection or pool.get_connection(command_name, **options)
   1171 
   1172         try:

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in get_connection(self, command_name, *keys, **options)
   1315             # closed. either way, reconnect and verify everything is good.
   1316             try:
-> 1317                 if connection.can_read():
   1318                     raise ConnectionError("Connection has data")
   1319             except ConnectionError:

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    793         if not sock:
    794             self.connect()
--> 795         return self._parser.can_read(timeout)
    796 
    797     def read_response(self, disable_decoding=False):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    315 
    316     def can_read(self, timeout):
--> 317         return self._buffer and self._buffer.can_read(timeout)
    318 
    319     def read_response(self, disable_decoding=False):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    222 
    223     def can_read(self, timeout):
--> 224         return bool(self.length) or self._read_from_socket(
    225             timeout=timeout, raise_on_timeout=False
    226         )

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in _read_from_socket(self, length, timeout, raise_on_timeout)
    192                 sock.settimeout(timeout)
    193             while True:
--> 194                 data = self._sock.recv(socket_read_size)
    195                 # an empty string indicates the server shutdown the socket
    196                 if isinstance(data, bytes) and len(data) == 0:

ConnectionResetError: [Errno 54] Connection reset by peer

Running it second time works perfectly, though it should have taken care by Retry strategy.

client.keys()
# prints all keys

configuration

redis server - 6.2.6
python redis - 4.1.0

We can write our own try/catch around redis client, but we are using some libraries like rq and flask-cache which uses redis internally and have no interface to modify it's flow.

Any help is much appreciated.

Irmine

Answer 1

I end up writing a custom class of Redis which retries on failure. Sharing it here in case anyone finds it useful.

import redis
import logging

from retry import retry

logger = logging.getLogger(__name__)


class RedisCustom(redis.Redis):
    def __init__(self, *args, **kwargs):
        super(RedisCustom, self).__init__(*args, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def set(self, key, value, ex=None, px=None, nx=False, xx=False):
        return super(RedisCustom, self).set(key, value, ex, px, nx, xx)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def get(self, name):
        return super(RedisCustom, self).get(name)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def delete(self, *names):
        return super(RedisCustom, self).delete(*names)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hset(self, name, key=None, value=None, mapping=None):
        return super(RedisCustom, self).hset(name, key, value, mapping)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hmset(self, name, mapping):
        return super(RedisCustom, self).hmset(name, mapping)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hgetall(self, name):
        return super(RedisCustom, self).hgetall(name)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def scan_iter(self, match=None, count=None, _type=None, **kwargs):
        return super(RedisCustom, self).scan_iter(match, count, _type, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hdel(self, name, *keys):
        return super(RedisCustom, self).hdel(name, *keys)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def keys(self, pattern="*", **kwargs):
        return super(RedisCustom, self).keys(pattern, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def info(self, section=None, **kwargs):
        return super(RedisCustom, self).info(section, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def ping(self, **kwargs):
        return super(RedisCustom, self).ping(**kwargs)

Irmine

Answer 2

You can also set a healtcheck value,
client = redis.Redis(..., health_check_interval=30)
it will reestablish the connection. More details in https://github.com/redis/redis-py/issues/1186

Korey

Answer 3

Try using a connection Pool as it will keep alive and automatically attempt to recover in the background. In the REPL

import redis
pool = redis.ConnectionPool(host="localhost", port=6379)
r = redis.Redis(connection_pool=pool)
r.ping()

In another shell you can test by running redis-cli

client LIST
client kill IP:PORT

Then run the

r.ping

And you should see a new connection in the redis-cli

client LIST

Menispermaceous

It's not working with RESET event. And redis internally uses pool by default if not specified. – Irmine