Celery: Disable heartbeat between workers

Question

I'm working on a data processing routine in Celery with a Redis backend and broker. Many workers (~200) interact with a broker to get tasks and execute those tasks. However, my workers are all sending heartbeat signals to one another, which populates their logs with all sorts of cruft like this:

[2018-05-13 15:38:00,737: INFO/MainProcess] missed heartbeat from [email protected]
[2018-05-13 15:38:00,737: INFO/MainProcess] missed heartbeat from [email protected]
[2018-05-13 15:38:00,737: INFO/MainProcess] missed heartbeat from [email protected]
[2018-05-13 15:38:00,738: INFO/MainProcess] missed heartbeat from [email protected]
[2018-05-13 15:38:00,738: INFO/MainProcess] missed heartbeat from [email protected]
[2018-05-13 15:38:00,739: INFO/MainProcess] missed heartbeat from [email protected]

The workers should never interface with each other directly, they should all get the information they need from the broker. Is it possible to disable heartbeats between worker nodes? If so, is this a bad idea for some reason I'm not yet seeing?

Legislative

Doesn't your log say that MainProcess missed a heartbeat from some worker? Not another worker, but a Main. – Lasley

Answer 1

You can try running your workers with --without-gossip to prevent this from happening. Starting in Celery 3.1, workers became passively subscribed to other worker events like heartbeats.

Gossip was added to allow celery users to take advantage of worker communication, like rerouting tasks to the best worker, but it is fine to disable this if there is no reason for the workers to communicate. You can read more about what worker gossip is/why it was introduced here: Celery 3.1 What's New.

You might also add the --without-mingle option to disable worker synchronization on startup.

Also, this question seems related: celery missed heartbeat (on_node_lost)

Sarge

@BenchurWong I am facing same issue ( INFO/MainProcess] missed heartbeat from worker@ip- ), What is the other possible ripple/issue if I do not turn this off (e.g set this value --without-gossip)??? as I do not have any problem with this message if it stays in my celery log files. – Agonistic

@Sarge can you please share feedback on my above comment. – Agonistic

@WaqasAli 😄I have to run two task and one worker in each task in Celery, this my scene. My solution is: 1. unique task queue; 2. only one worker. So you can point like this: --pool=solo -n [worker-name] --without-gossip --without-mingle -Q [queue-name] – Ballman