Send log messages from all celery tasks to a single file

Question

I'm wondering how to setup a more specific logging system. All my tasks use

logger = logging.getLogger(__name__)

as a module-wide logger.

I want celery to log to "celeryd.log" and my tasks to "tasks.log" but I got no idea how to get this working. Using CELERYD_LOG_FILE from django-celery I can route all celeryd related log messages to celeryd.log but there is no trace of the log messages created in my tasks.

Puckett

Answer 1

Note: This answer is outdated as of Celery 3.0, where you now use get_task_logger() to get your per-task logger set up. Please see the Logging section of the What's new in Celery 3.0 document for details.

---

Celery has dedicated support for logging, per task. See the Task documentation on the subject:

You can use the workers logger to add diagnostic output to the worker log:

@celery.task()
def add(x, y):
    logger = add.get_logger()
    logger.info("Adding %s + %s" % (x, y))
    return x + y

There are several logging levels available, and the workers loglevel setting decides whether or not they will be written to the log file.

Of course, you can also simply use print as anything written to standard out/-err will be written to the log file as well.

Under the hood this is all still the standard python logging module. You can set the CELERYD_HIJACK_ROOT_LOGGER option to False to allow your own logging setup to work, otherwise Celery will configure the handling for you.

However, for tasks, the .get_logger() call does allow you to set up a separate log file per individual task. Simply pass in a logfile argument and it'll route log messages to that separate file:

@celery.task()
def add(x, y):
    logger = add.get_logger(logfile='tasks.log')
    logger.info("Adding %s + %s" % (x, y))
    return x + y 

Last but not least, you can just configure your top-level package in the python logging module and give it a file handler of it's own. I'd set this up using the celery.signals.after_setup_task_logger signal; here I assume all your modules live in a package called foo.tasks (as in foo.tasks.email and foo.tasks.scaling):

from celery.signals import after_setup_task_logger
import logging

def foo_tasks_setup_logging(**kw):
    logger = logging.getLogger('foo.tasks')
    if not logger.handlers:
        handler = logging.FileHandler('tasks.log')
        formatter = logging.Formatter(logging.BASIC_FORMAT) # you may want to customize this.
        handler.setFormatter(formatter)
        logger.addHandler(handler)
        logger.propagate = False

after_setup_task_logger.connect(foo_tasks_setup_logging)

Now any logger whose name starts with foo.tasks will have all it's messages sent to tasks.log instead of to the root logger (which doesn't see any of these messages because .propagate is False).

Tabling

Are the log messages buffered or unbuffered? I'm wondering whether out-of-order log messages is an indication of tasks executing out of order. – Lepidosiren

@EricWalker: logging doesn't buffer anything. FileHandler uses a regular open() call, and the default is to open the file in textmode so writing to that will use line buffering (a flush after every newline, which means every log entry). – Tabling

Seems there is a typo in 'CELERYD_HIJACK_ROOT_LOGGER' (and not 'CELERY_HIJACK_ROOT_LOGGER') – Carotid

@imbolc: I can't find any references to CELERY_WORKER_HIJACK_ROOT_LOGGER, not sure what you are saying here. There is only CELERYD_HIJACK_ROOT_LOGGER and the worker_hijack_root_logger configuration option name (the latter is the 4.x lowercase version of the former). – Tabling

@MartijnPieters yep, it's about 4.x, you just have to make it prefixed and upper cased to use in settings.py – Briannebriano

@imbolc: this answer is about a much older Celery release, however. Nothing in it applies to Celery 4. – Tabling

@MartijnPieters I agree it's not likely TS was asking about future releases :) But people coming here nowadays are probably working with modern versions. – Briannebriano

@imbolc: yes, and for those versions you'd use get_task_logger, as stated at the top. – Tabling

Answer 2

Just a hint: Celery has its own logging handler:

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)

Also, Celery logs all output from the task. More details at Celery docs for Task Logging

Eclogue

Answer 3

join --concurrency=1 --loglevel=INFO with the command to run celery worker

eg: python xxxx.py celery worker --concurrency=1 --loglevel=INFO

Better to set loglevel inside each python files too

Bacolod