Python daemon and systemd service

Question

I have a simple Python script working as a daemon. I am trying to create systemd script to be able to start this script during startup.

Current systemd script:

[Unit]
Description=Text
After=syslog.target

[Service]
Type=forking
User=node
Group=node
WorkingDirectory=/home/node/Node/
PIDFile=/var/run/zebra.pid
ExecStart=/home/node/Node/node.py

[Install]
WantedBy=multi-user.target

node.py:

if __name__ == '__main__':
    with daemon.DaemonContext():
        check = Node()
        check.run()

run contains while True loop.

I try to run this service with systemctl start zebra-node.service. Unfortunately service never finished stating sequence - I have to press Ctrl+C. Script is running, but status is activating and after a while it change to deactivating. Now I am using python-daemon (but before I tried without it and the symptoms were similar).

Should I implement some additional features to my script or is systemd file incorrect?

Poise

Did the answer solve your problem? If not, try setting daemon_context=True when creating the DaemonContext(). It may work. – Circumference

@Poise It is pity, your Python code example is not complete (missing import of daemon and not clear, where the Node comes from) so it is not easy/possible to reproduce your situation. – Lyingin

@Poise This indirectly related to the question, but might help you: unix.stackexchange.com/a/226853/33386 – Gorgonian

Answer 1

The reason, it does not complete the startup sequence is, that for Type forking your startup process is expected to fork and exit (see $ man systemd.service - search for forking).

Simply use only the main process, do not daemonize

One option is to do less. With systemd, there is often no need to create daemons and you may directly run the code without daemonizing.

#!/usr/bin/python -u
from somewhere import Node
check = Node()
check.run()

This allows using simpler Type of service called simple, so your unit file would look like.

[Unit]
Description=Simplified simple zebra service
After=syslog.target

[Service]
Type=simple
User=node
Group=node
WorkingDirectory=/home/node/Node/
ExecStart=/home/node/Node/node.py
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

Note, that the -u in python shebang is not necessary, but in case you print something out to the stdout or stderr, the -u makes sure, there is no output buffering in place and printed lines will be immediately caught by systemd and recorded in journal. Without it, it would appear with some delay.

For this purpose I added into unit file the lines StandardOutput=syslog and StandardError=syslog. If you do not care about printed output in your journal, do not care about these lines (they do not have to be present).

systemd makes daemonization obsolete

While the title of your question explicitly asks about daemonizing, I guess, the core of the question is "how to make my service running" and while using main process seems much simpler (you do not have to care about daemons at all), it could be considered answer to your question.

I think, that many people use daemonizing just because "everybody does it". With systemd the reasons for daemonizing are often obsolete. There might be some reasons to use daemonization, but it will be rare case now.

EDIT: fixed python -p to proper python -u. thanks kmftzg

Lyingin

The reason to daemonize is to support other platforms that don't use systemd. Having to make a separate code path for systemd is another way systemd inhibits portability. – Interdenominational

@NickBastin and this is another way portability inhibits progress & simplification. – Siobhan

@intelfx: I'm sure all the developers who wrote apps relying on the progress and simplification of OS/2 agree with you. – Interdenominational

@NickBastin systemd does support forking applications so there is no need for separate codepath for systemd, if you do not want to. Not reading documentation inhibits practical skills and can make comments baseless. – Lyingin

@JanVlcinsky Yes but your answer doesn't suggest this option - it tells the OP to change their code instead of telling them how to support systemd without sacrificing support for non-systemd platforms. Your own answer suggests "There might be some reasons to use daemonization, but it will be rare case now." and I was merely commenting originally on why you should daemonize, since you appeared to miss the point. – Interdenominational

@NickBastin The OP talks about "simple python script" and use of "systemd". No request for non-systemd platforms portability, such request exists only in your reactions blaming systemd. – Lyingin

FYI – -p isn't a thing, but -u is. See docs.python.org/3/using/cmdline.html#cmdoption-u – Mccombs

There are clearly other process management tools, such as supervisord or pm2. I thinking "not-to-daemonize" is clearly the unix way of doing things. – Pus

I've a working python daemon using python-daemon standard pattern (based in the one described here: dpbl.wordpress.com/2017/02/12/a-tutorial-on-python-daemon) and is works pretty well with and without systemd. I just comment out the systemd Type value to leave it the default. systemd start/stop works as expected: signal listen, as well as running it by hand with --daemon process: it checks pid and other important stuff, so you can daemonize a python script with few lines and make it systemd compatible. I use systemd to run on startup and restart on really unexpected errors. – Euphemia

I did exactly the same unit file and everything goes fine apart from getting error on startup after reboot. It says there is a cyclic dependency as: Job active_controller.service/start deleted to break ordering cycle starting with sysinit.t arget/start – Lewd

Is not this because I have a {while True:} loop in my script? – Lewd

Answer 2

It is possible to daemonize like Schnouki and Amit describe. But with systemd this is not necessary. There are two nicer ways to initialize the daemon: socket-activation and explicit notification with sd_notify().

Socket activation works for daemons which want to listen on a network port or UNIX socket or similar. Systemd would open the socket, listen on it, and then spawn the daemon when a connection comes in. This is the preferred approch because it gives the most flexibility to the administrator. [1] and [2] give a nice introduction, [3] describes the C API, while [4] describes the Python API.

[1] http://0pointer.de/blog/projects/socket-activation.html
[2] http://0pointer.de/blog/projects/socket-activation2.html
[3] http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html
[4] http://www.freedesktop.org/software/systemd/python-systemd/daemon.html#systemd.daemon.listen_fds

Explicit notification means that the daemon opens the sockets itself and/or does any other initialization, and then notifies init that it is ready and can serve requests. This can be implemented with the "forking protocol", but actually it is nicer to just send a notification to systemd with sd_notify(). Python wrapper is called systemd.daemon.notify and will be one line to use [5].

[5] http://www.freedesktop.org/software/systemd/python-systemd/daemon.html#systemd.daemon.notify

In this case the unit file would have Type=notify, and call systemd.daemon.notify("READY=1") after it has established the sockets. No forking or daemonization is necessary.

Margalo

Looks good. How to install this python library which provides systemd.daemon via pip? – Intemperate

Official installation instructions at github.com/systemd/python-systemd#installation show how to install with pip. If they don't work for you, please file an issue at github.com/systemd/python-systemd/issues. – Margalo

Answer 3

You're not creating the PID file.

systemd expects your program to write its PID in /var/run/zebra.pid. As you don't do it, systemd probably thinks that your program is failing, hence deactivating it.

To add the PID file, install lockfile and change your code to this:

import daemon
import daemon.pidlockfile 

pidfile = daemon.pidlockfile.PIDLockFile("/var/run/zebra.pid")
with daemon.DaemonContext(pidfile=pidfile):
    check = Node()
    check.run()

(Quick note: some recent update of lockfile changed its API and made it incompatible with python-daemon. To fix it, edit daemon/pidlockfile.py, remove LinkFileLock from the imports, and add from lockfile.linklockfile import LinkLockFile as LinkFileLock.)

Be careful of one other thing: DaemonContext changes the working dir of your program to /, making the WorkingDirectory of your service file useless. If you want DaemonContext to chdir into another directory, use DaemonContext(pidfile=pidfile, working_directory="/path/to/dir").

Tribble

The last paragraph about how DaemonContext changes the working directory of the program just solved my daemonzing problems – Solarium

Since less code is more, I prefer the answer "Simply use only the main process, do not daemonize". – Intemperate

idk if the API has changed in Python3. but there it must be import daemon.pidfile and not import daemon.pidlockfile – Seafaring

when the lock is removed it always remains locked even the daemon completed – Jerri

Answer 4

I came across this question when trying to convert some python init.d services to systemd under CentOS 7. This seems to work great for me, by placing this file in /etc/systemd/system/:

[Unit]
Description=manages worker instances as a service
After=multi-user.target

[Service]
Type=idle
User=node
ExecStart=/usr/bin/python /path/to/your/module.py
Restart=always
TimeoutStartSec=10
RestartSec=10

[Install]
WantedBy=multi-user.target

I then dropped my old init.d service file from /etc/init.d and ran sudo systemctl daemon-reload to reload systemd.

I wanted my service to auto restart, hence the restart options. I also found using idle for Type made more sense than simple.

Behavior of idle is very similar to simple; however, actual execution of the service binary is delayed until all active jobs are dispatched. This may be used to avoid interleaving of output of shell services with the status output on the console.

More details on the options I used here.

I also experimented with keeping the old service and having systemd resart the service but I ran into some issues.

[Unit]
# Added this to the above
#SourcePath=/etc/init.d/old-service 

[Service]
# Replace the ExecStart from above with these
#ExecStart=/etc/init.d/old-service start
#ExecStop=/etc/init.d/old-service stop

The issues I experienced was that the init.d service script was used instead of the systemd service if both were named the same. If you killed the init.d initiated process, the systemd script would then take over. But if you ran service <service-name> stop it would refer to the old init.d service. So I found the best way was to drop the old init.d service and the service command referred to the systemd service instead.

Hope this helps!

Federalize

Answer 5

Also, you most likely need to set daemon_context=True when creating the DaemonContext().

This is because, if python-daemon detects that if it is running under a init system, it doesn't detach from the parent process. systemd expects that the daemon process running with Type=forking will do so. Hence, you need that, else systemd will keep waiting, and finally kill the process.

If you are curious, in python-daemon's daemon module, you will see this code:

def is_detach_process_context_required():
    """ Determine whether detaching process context is required.

        Return ``True`` if the process environment indicates the
        process is already detached:

        * Process was started by `init`; or

        * Process was started by `inetd`.

        """
    result = True
    if is_process_started_by_init() or is_process_started_by_superserver():
        result = False

Hopefully this explains better.

Circumference

I could be mistaken, but I think this flag is called "detach_process" not and not "daemon_context" – Roorback