Celery/CloudAMQP error in a Heroku Flask App

Question

I'm running a Flask app on Heroku (on the free tier) and running into some trouble when scheduling tasks using apply_async. If I schedule more than two tasks, I get a long stacktrace with the exception:

AccessRefused(403, u"ACCESS_REFUSED - access to exchange 'celeryresults' in vhost 'rthtwchf' refused for user 'rthtwchf'", (40, 10), 'Exchange.declare')

The odd thing is the first two tasks (before restarting all of my processes) always seem to complete with no issue.

A little bit of search engine sleuthing leads me to https://mcmap.net/q/1169164/-celery-cannot-connect-remote-worker-with-new-username which makes it looks like a permissions issue, but I'd assume that the Heroku CloudAMPQ service would have taken care of that already.

Any advice is appreciated!

Poyssick

Let us have a look at it, email [email protected] – Raving

@CarlHörberg, public update? Experiencing this issue too, and judging by the upvotes there may be others. – Sennet

You're running out of connections. Make sure that you limit the connection pool and the number of workers. – Raving

Have you tried tweaking celery's BROKER_POOL_LIMIT to be 1 as described in the CloudAMQP docs? You could also try reducing the concurrency of celery as suggested with CELERYD_OPTS="--concurrency=1". This could prevent you from going over usage limits which might be your issue? – Johnajohnath

Do you really need to use the results backend? – Flybynight

Answer 1

I think your connections are exceeding 3 (free plan limit). Set the BROKER_POOL_LIMIT to 1 and it will work.

Reportorial