Heroku Celery Exceeding Connection Limit of 6 when Broker Pool Limit > None

Question

I am using Django/Celery on Heroku with the CloudAMQP add-on.

In my settings file, if I set BROKER_POOL_LIMIT to any value other than None, I quickly exceed my CloudAMQP connection limit of 6.

I'm wondering why I would have more connections than what I specify in BROKER_POOL_LIMIT and how can I set BROKER_POOL_LIMIT to a value greater than None without eventually exceeding the connection limit. In case it's useful, this is my worker process type as specified in my profile

worker python manage.py celery worker -B --loglevel=info

Inlier

How many gunicorn web-workers are you spawning? – Chet

Each gunicorn web-worker (-w) will spawn a connection pool, so the formula is: BROKER_POOL_LIMIT * (gunicorn-workers * web dynos + worker dynos) – Chet

I see. I am running 8 web-workers on 2 web dynos. Not that I am sure this is the optimal configuration. I also have 1 worker dyno for running background tasks using celery. So your saying that with this set up, if I set BROKER_POOL_LIMIT to anything other than None I will exceed the 6 concurrent connection limit I get with the Tough Tiger CloudAMQP add on? – Inlier

Yes correct, but if you decrease to 2 web workers per dyno you should be fine, 1 * ( 2 * 2 + 1) = 5 – Chet

@CarlHörberg The formula for what exactly? Did you mean to write BROKER_POOL_LIMIT = instead of * maybe? Also, shouldn't the last part of the formula be worker dynos * concurrency? Would you be willing to post a more detailled version of your comment as an answer to this question maybe? – Nekton

@HenrikHeimbuerger I read it as connections = BROKER_POOL_LIMIT * (gunicorn-workers * web dynos + worker dynos). But I second turning the comment into an answer. – Spang