Why do I have so many Celery messages with CloudAMQP on Heroku?

Question

My Django site running on Heroku is using CloudAMQP to handle its scheduled Celery tasks. CloudAMQP is registering many more messages than I have tasks, and I don't understand why.

e.g., in the past couple of hours I'll have run around 150 scheduled tasks (two that run once a minute, another that runs once every five minutes), but the CloudAMQP console's Messages count increased by around 1,300.

My relevant Django settings:

BROKER_URL = os.environ.get("CLOUDAMQP_URL", "")
BROKER_POOL_LIMIT = 1
BROKER_HEARTBEAT = None
BROKER_CONNECTION_TIMEOUT = 30
CELERY_ACCEPT_CONTENT = ['json',]
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_RESULT_EXPIRES = 7 * 86400
CELERY_SEND_EVENTS = False
CELERY_EVENT_QUEUE_EXPIRES = 60
CELERY_RESULT_BACKEND = None
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

My Procfile:

web: gunicorn myproject.wsgi --log-file -
main_worker: python manage.py celery worker --beat --without-gossip --without-mingle --without-heartbeat --loglevel=info

Looking at the Heroku logs I only see the number of scheduled tasks running that I'd expect.

The RabbitMQ overview graphs tend to look like this most of the time:

I don't understand RabbitMQ enough to know if other panels shed light on the problem. I don't think they show anything obvious that would account for all these messages.

I'd like to at least understand what the extra messages are, and then whether there's a way I can eliminate some or all of them.

Underestimate

You can always email [email protected] with your username and we'll take a look at it – Lisabeth

Thanks Carl. I get so used to using Stack Overflow I sometimes forget that some places actually have support :) – Underestimate

Answer 1

I'had got the same error few days ago. For those who get the same issue, CloudAMQP's doc recommend to add some arguments when you launch your celery :

--without-gossip --without-mingle --without-heartbeat

"This will decrease the message rates substantially. Without these flags Celery will send hundreds of messages per second with different diagnostic and redundant heartbeat messages."

And indeed, this fix the issue so far ! And you finally get only YOUR messages sent.

Stapleton