Celery, Redis and ConnectionPool

Question

I recently had a problem with Celery and Redis : I was using too many connections to my cloud Redis account.

I seem to have fixed the problem for now with better settings and a bigger Redis plan.

However, it lead me to some experimentation on how Redis and Celery work together, and there is something I don't understand : the number of Redis ConnectionPool that are created by the Python Redis module.

I configured Celery to use Redis as a broker and a result backend.

Here is my Celery config :

CELERY_TIMEZONE = 'Europe/Paris'
CELERY_BROKER_URL = REDIS_URL
CELERY_RESULT_BACKEND = REDIS_URL
CELERY_TASK_SERIALIZER = 'pickle'
CELERY_SEND_EVENTS = False
CELERY_IMPORTS = ('task1', 'task2')
CELERY_ACCEPT_CONTENT = ['pickle']
CELERY_REDIS_MAX_CONNECTIONS = 2
BROKER_POOL_LIMIT = 1
BROKER_HEARTBEAT = None
BROKER_TRANSPORT_OPTIONS = {
   'max_connections': 30
}

I run Celery with the following command :

celery worker --without-gossip --without-mingle --without-heartbeat --concurrency=1 --app=backZest.celery

Right after Celery has booted, I already have 10 connections to my Redis instance : is that normal?

I checked how many redis ConnectionPool were created : to do this, I just modified the Python Redis module connection.py file, in the __init__ method of the ConnectionPool class:

import sys
print "ConnectionPool instance %s with %s max_connections" % (id(self), self.max_connections)
sys.stdout.flush()

As you can see, there are many (8) pools created, all of them with the same max_connections, 30, which is my setting in BROKER_TRANSPORT_OPTIONS :

ConnectionPool instance 4412957392 with 30 max_connections
ConnectionPool instance 4412959888 with 30 max_connections
ConnectionPool instance 4420369616 with 30 max_connections
ConnectionPool instance 4420370320 with 30 max_connections
ConnectionPool instance 4420367056 with 30 max_connections
ConnectionPool instance 4420491792 with 30 max_connections
ConnectionPool instance 4422318224 with 30 max_connections
ConnectionPool instance 4422319504 with 30 max_connections

I don't understand why there are so many of them. I want to control the number of connections to my Redis. I can control the number of max_connections per ConnectionPool but isn't it a bit useless if I don't control the number of ConnectionPool created ?

Thanks a lot for your help !

Atropine

If I remember well, whenever you instantiate Redis class it creates a new ConnectionPool (!) so it is the best (if you can) to re-use the same instance... Maybe new Python redis client does not do this any-more... – Aftonag

Answer 1

Yes, this is normal. Pool creates connections beforehand so they are ready when you need them. Establishing a connection may take a lot of time, so pool helps you save on that. Use max_connections to restrict the pool as you need it. And don't worry that pool opens connections early.

Aporia