celery: "Substantial drift from"

Question

I have quite a problem with celery on my distribted system. I have couple of machines among different localizations and I've got a lot of warnings in my log files like:

"Substantial drift from celery@host [...]"

I was able to set date to return the same values (even that the machines are in different countries) but python print(utcoffset()) returns different results on main server and nodes.

How to fix that issue? I was unable to find any good solution except that utcoffset() should return the same value - but how?

Thanks for help.

Willie

Do you have any code to show us what you're doing/doing wrong? – Weaver

utcoffset() returns 2 on one machine and 3 on second – Willie

Ask this question over at serverfault.com There, people will be able to tell you how to set up a system to keep your server clocks largely in line. This is a non-trivial undertaking but there are good solutions out there. – Weaver

Also make sure that you're running celery 3.1.7, as there was a bug in the calculation of the utc offset on earlier versions. – Chabot

Answer 1

I encountered a similar issue while starting flower:

[W 161208 08:42:00 state:74] Substantial drift from [email protected] may mean clocks are out of sync.  Current drift is 10800 seconds.  [orig: 2016-12-08 08:42:00.722560 recv: 2016-12-08 05:42:00.719938]

versions used:

celery==3.1.23 (and 3.1.25)
flower==0.9.1
rabbitmq 3.5.6

with settings:

CELERY_TIMEZONE='US/Pacific'
TIME_ZONE = 'US/Pacific'
USE_TZ = True

Looked through all available issue reports and suggestions on stackoverflow and git, but all indicated fixes that I should already have in place with > 3.1.23

This is how I solved it

My system and subsequent bash environment was running EST for timezone. As you can see above, the django app runs 'US/Pacific' (PST). This is is a 3 hour difference, or 10800 seconds.

By setting the bash timezone to PST as well, to match the django app, the drift error went away:

export TZ="US/Pacific"

Sunfish

Answer 2

As long as you use CELERY_ENABLE_UTC having the same utcoffset() is not a requirement.

Are you hitting issue #1112? You might need to update to Celery 3.1.

Serles

Answer 3

While configuring Airflow to use RabbitMQ, set the value as follows in airflow.cfg:

[celery_broker_transport_options]
 consumer_timeout = 31622400000

And execute the following command with a running rabbitmq server

rabbitmqctl eval 'application:set_env(rabbit, consumer_timeout, 36000000).'

Brit