Running celery as daemon does not create PID file

Question

I have been scratching my brains on this one since past few days, I have seen other issues on stackoverflow (as it is a duplicate question) and I have tried everything to make this work, the workers are running fine but the celery is not starting up as a process.

I run the command:

sudo service celeryd start

and I get:

celery init v10.1.
Using config script: /etc/default/celeryd
celery multi v3.1.23 (Cipater)
> Starting nodes...
    > worker1@ip-172-31-21-215: OK

I run:

sudo service celeryd status

and I get:

celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found

The celeryd down: no pidfiles found error is what I need to resolve.

I know this question is a duplicate one but still go along with me on this one because I have tried all of them and still unable to get it resolved.

I am deploying this script on Amazon Web Services. I am using a virtual environment.

The init.d script is taken directly from the here and then I gave it the required permissions.

Here is my configuration file:

# Names of nodes to start
#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS (see `celery multi --help` for examples):
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10

# Absolute or relative path to the 'celery' command:
# CELERY_BIN="/usr/local/bin/celery"
CELERY_BIN="/home/<user>/.virtualenvs/<virtualenv_name>/bin/celery"

# App instance to use
# comment out this line if you don't use an app
# CELERY_APP="proj"
# or fully qualified:
CELERY_APP="<project_name>.settings:app"

# Where to chdir at start.
CELERYD_CHDIR="/home/<user>/projects/<project_name>/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists, e.g. nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

I have used the process to create the celery user using this article.

My project is a Django project and I have specified the DJANGO_SETTINGS_MODULE environment variable in the celery setting file as specified in the documentation and also in the stackoverflow answer.

Do I need to change anything in the init.d script or anything else that needs to be added in the celery configuration file... Is it about the celery user that I have created because I also tried specifying

CELERYD_USER = ""
CELERYD_GROUP = ""

while also changing the DEFAULT_USER value to "" in the init.d script. Still the issue persisted.

In one of the answers it was also told that there might be some errors in the project... but I did not find any such errors all thanks to my test cases.

PS : I have specified , and for privacy issues they have their original names.

Swanson

I've got the same issue. It seems following the documentation leads you here sometimes. – Equi

Updating my celery version from 4.1 and 4.2 in my virtual environment seems to have fixed the issue. – Sharpnosed

Answer 1

I was having this a similar issue on my ubuntu server [ERROR 2]FILE NOT FOUND. Turns out, /var/run/celery/ Directories don't get automatically created even if you set that in the celery.service configuration done in the celery example docs. You can make that directory, and grant the right permissions manually, but as soon you reboot the server the directory will vanish because it's in a temporary directory. After some reading about how the linux system operates, I found out you just need to create a configuration file in /etc/tmpfiles.d/celery.conf with these lines

d /var/run/celery 0755 admin admin -
d /var/log/celery 0755 admin admin -                                       

Note: you will need to use a different user:group other than 'admin' or you can create a user:group called admin specifically to handle your celery process.

You can read more about this configuration and the way it operates by typing

man tmpfiles.d

Erratum

I don't believe you need the second line – Variform

Thank you! I have a user called "web" and I used it instead of "admin". That fixed my issue of the file disappearing when I reboot. I also learned a bunch about tmpfiles.d. Interesting stuff. Anyways, thank you! – Wanitawanneeickel

Answer 2

I had the issue and solved it just now, thank god! For me it was a permission issue. I had expected it to be in /var/run/celery or /var/log/celery but it turns out it was the log file I have setup Django logging for. For some reason celery wanted to write to that file (I have to look into that) but had no permission. I found the error with the verbose command and skip daemonization step:

# C_FAKEFORK=1 sh -x /etc/init.d/celeryd start 

This is an old thread but if anyone of you run into this error, I hope this may help!

Good luck!

Nasho

Answer 3

I saw the same issue and it turned out to be a permissions issue. Make sure to set the user/group that celery is running under to own the /var/log/celery/ and /var/run/celery/ folders. See here for a step by step example: Daemonizing celery

Peripheral