celery daemon - permission denied on log file

Question

I have been working on setting up my celery task as a daemon in order to process data on a schedule.

I have been following the docs in order to set up my daemon, but have been running into a log file permission error that has me stumped.

Below is the configuration i have set up on an ubuntu box on Digital Ocean

/etc/default/celeryd

# here we have a single node
CELERYD_NODES="w1"

CELERY_BIN = "/mix_daemon/venv/bin/celery"
CELERYD_CHDIR="/mix_daemon/"

CELERYD_OPTS="-A tasks worker --loglevel=info --beat"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERY_CREATE_RUNDIR=0
CELERY_CREATE_LOGDIR=0

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

however, when i run

sh -x /etc/init.d/celeryd start

i get the following error message:

celery multi v3.1.7 (Cipater)
> Starting nodes...
> celery@mix: OK
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/john/mix_daemon/venv/lib/python2.7/site-packages/celery/bin/celeryd_detach.py", line 168, in <module> main()
File "/home/john/mix_daemon/venv/lib/python2.7/site-packages/celery/bin/celeryd_detach.py", line 165, in main
detached_celeryd(app).execute_from_commandline()
File "/home/john/mix_daemon/venv/lib/python2.7/site-packages/celery/bin/celeryd_detach.py", line 160, in execute_from_commandline
**vars(options)
File "/home/john/mix_daemon/venv/lib/python2.7/site-packages/celery/bin/celeryd_detach.py", line 42, in detach
with detached(logfile, pidfile, uid, gid, umask, working_directory, fake):
File "/home/john/mix_daemon/venv/local/lib/python2.7/site-packages/celery/platforms.py", line 314, in open
self.after_chdir()
File "/home/john/mix_daemon/venv/local/lib/python2.7/site-packages/celery/platforms.py", line 384, in after_chdir_do
logfile and open(logfile, 'a').close()
IOError: [Errno 13] Permission denied: '/var/log/celery/celery.log'
+ sleep 5
+ exit 0

I looked this error message up and saw this answer from a redmine project. So i tried the following to try to allow for the celery worker to write to the log file:

$ sudo mkdir -p -m 2755 /var/log/celery
$ sudo chown celery:celery /var/log/celery

but the same error remains when i try to start the daemon.

I am a celery noob, and any help on this would be greatly appreciated!

Paleozoology

https://mcmap.net/q/831588/-permission-problems-prevent-celery-from-running-as-daemon – Nibelung

Answer 1

According to the documentation:

The init scripts can only be used by root, and the shell configuration file must also be owned by root"

So, you need to run:

sudo chown root:root /etc/default/celeryd

and

sudo /etc/init.d/celeryd start

By the way, you have invalid syntax in your celeryd config file:

CELERY_BIN = "/mix_daemon/venv/bin/celery"

spaces is not allowed with "=" operator, the correct is:

CELERY_BIN="/mix_daemon/venv/bin/celery"

Also check, that you have your bin in /mix_daemon not /home//mix_daemon

Uxorial

Answer 2

It helps me:

sudo chown -R myuser:root /var/run/celery/
sudo chown -R myuser:root /var/log/celery/

Maffa

This was necessary for me when using systemd, where myuser is whatever the User is set to in the celery.service file. – Rhodesia

Answer 3

Tl;dr

If you're using systemd for daemonization, your file /etc/tmpfiles.d/celery.conf should look like this:

d /run/celery 0755 myuser myusergroup -
d /var/log/celery 0755 myuser myusergroup -

A little more detail

For those using systemd, I found that I'd been using the wrong user when creating working directories for celery's logs and once I'd updated /etc/tmpfiles.d/celery.conf with the correct user and user group, my issues were solved.

The documentation mentions:

You can also use systemd-tmpfiles in order to create working directories (for logs and pid).

file: /etc/tmpfiles.d/celery.conf

And gives the example:

d /run/celery 0755 celery celery -
d /var/log/celery 0755 celery celery -

Your file should look like this:

d /run/celery 0755 myuser myusergroup -
d /var/log/celery 0755 myuser myusergroup -

Leash