I'm using a redis docker container to be used as a message/ broker queue for celery and flaskk-socketio in my python app and lately my redis instance is crashing pretty often, and it doesn't come back.
I'm not a redis expert and I couldn't yet find what the problem is, here are the logs, I appreciate any help!
Thanks
[2019-08-09 03:44:29,930: CRITICAL/MainProcess] Unrecoverable error: ResponseError('UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)',)
Traceback (most recent call last):
File "/server/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
blueprint.start(self)
File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 598, in start
c.loop(*c.loop_args())
File "/server/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop
connection.drain_events(timeout=2.0)
File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 315, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
get(self._deliver, timeout=timeout)
File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 370, in get
ret = self.handle_event(fileno, event)
File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 352, in handle_event
return self.on_readable(fileno), self
File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 348, in on_readable
chan.handlers[type]()
File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 725, in _brpop_read
**options)
File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
raise response
redis.exceptions.ResponseError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
[2019-07-15 07:04:53,803: WARNING/MainProcess] Traceback (most recent call last):
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait
listener.cb(fileno)
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/celery/worker/pidbox.py", line 122, in loop
pass
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 798, in __exit__
self.release()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 372, in release
self._close()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 338, in _close
self._do_close_self()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 331, in _do_close_self
self.transport.close_connection(self._connection)
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 953, in close_connection
channel.close()
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 851, in close
self.queue_delete(queue, client=client)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 542, in queue_delete
self._delete(queue, exchange, *meta, **kwargs)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 810, in _delete
queue or '']))
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 1661, in srem
return self.execute_command('SREM', name, *values)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 668, in execute_command
return self.parse_response(connection, command_name, **options)
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
raise response
[2019-07-15 07:04:53,806: WARNING/MainProcess] redis.exceptions.ReadOnlyError: You can't write against a read only replica.
[2019-07-15 07:04:53,806: WARNING/MainProcess] Removing descriptor: 19
redis.exceptions.ConnectionError: Error 113 connecting to outflink_redis:6379. EHOSTUNREACH.
– Casabonne