How do you create a daemon in Python?

Question

Searching on Google reveals x2 code snippets. The first result is to this code recipe which has a lot of documentation and explanation, along with some useful discussion underneath.

However, another code sample, whilst not containing so much documentation, includes sample code for passing commands such as start, stop and restart. It also creates a PID file which can be handy for checking if the daemon is already running etc.

These samples both explain how to create the daemon. Are there any additional things that need to be considered? Is one sample better than the other, and why?

Chipboard

I've always found daemonization code unneeded. Why not just let the shell do it? – Subsidy

Because it doesn't do setsid or setpgrp. – Ima

Use supervisord.org. This way you don't need to fork() or redirect you stdin/stderr. Just write a normal program. – Tusker

Answer 1

Current solution

A reference implementation of PEP 3143 (Standard daemon process library) is now available as python-daemon.

Historical answer

Sander Marechal's code sample is superior to the original, which was originally posted in 2004. I once contributed a daemonizer for Pyro, but would probably use Sander's code if I had to do it over.

Pneumatic

Edit: Since I originally posted this reply, a reference implementation of PEP 3143 in now available: pypi.python.org/pypi/python-daemon – Pneumatic

@JeffBauer Original link has died, I remember it being useful, you wouldn't happen to know a live link for that would you? – Natalia

@CrazyCasta: Sander Marechal's version is still available on the Wayback Machine – Pneumatic

@JeffBauer: Sander's code is still better than http://pypi.python.org/pypi/python-daemon. More reliable. Just one example : try to start two times the same daemon with python-daemon : big ugly error. With Sander's code : a nice notice "Daemon already running." – Pogrom

Yesterday, the Wayback Machine link worked, as of today it's broken. I can only assume that it's lost forever... – Cystocele

Since the "python-daemon" module documentation is still missing (see also many other SO questions) and is rather obscure (how to start/stop properly a daemon from command line with this module?), I modified Sander Marechal's code sample to add quit() method that gets executed before the daemon is stopped. Here it is. – Pogrom

Is there an alternative to the python-daemon which doesn't use the GPL-3.0 License? – Bernina

Answer 2

There are many fiddly things to take care of when becoming a well-behaved daemon process:

• prevent core dumps (many daemons run as root, and core dumps can contain sensitive information)

• behave correctly inside a chroot gaol

• set UID, GID, working directory, umask, and other process parameters appropriately for the use case

• relinquish elevated suid, sgid privileges

• close all open file descriptors, with exclusions depending on the use case

• behave correctly if started inside an already-detached context, such as init, inetd, etc.

• set up signal handlers for sensible daemon behaviour, but also with specific handlers determined by the use case

• redirect the standard streams stdin, stdout, stderr since a daemon process no longer has a controlling terminal

• handle a PID file as a cooperative advisory lock, which is a whole can of worms in itself with many contradictory but valid ways to behave

• allow proper cleanup when the process is terminated

• actually become a daemon process without leading to zombies

Some of these are standard, as described in canonical Unix literature (Advanced Programming in the UNIX Environment, by the late W. Richard Stevens, Addison-Wesley, 1992). Others, such as stream redirection and PID file handling, are conventional behaviour most daemon users would expect but that are less standardised.

All of these are covered by the PEP 3143 “Standard daemon process library” specification. The python-daemon reference implementation works on Python 2.7 or later, and Python 3.2 or later.

Bun

“gaol” is spelled correctly, because that's how W. Richard Stevens spelled it :-) – Bun

Gaol is an English thing. The poster is from Australia so it makes sense. – Townes

Any plans on making a py3k friendly version? – Mingo

Answer 3

Here's my basic 'Howdy World' Python daemon that I start with, when I'm developing a new daemon application.

#!/usr/bin/python
import time
from daemon import runner

class App():
    def __init__(self):
        self.stdin_path = '/dev/null'
        self.stdout_path = '/dev/tty'
        self.stderr_path = '/dev/tty'
        self.pidfile_path =  '/tmp/foo.pid'
        self.pidfile_timeout = 5
    def run(self):
        while True:
            print("Howdy!  Gig'em!  Whoop!")
            time.sleep(10)

app = App()
daemon_runner = runner.DaemonRunner(app)
daemon_runner.do_action()

Note that you'll need the python-daemon library. You can install it by:

pip install python-daemon

Then just start it with ./howdy.py start, and stop it with ./howdy.py stop.

Eared

That daemon module you import is not a standard part of Python (yet). It needs to be installed with pip install python-daemon or equivalent. – Cainozoic

I installed python-daemon as you described, but when I try to run my app (same as your last 3 lines), I get ImportError: cannot import name runner – Indisposition

Can you check if it's installed properly? $ dpkg -L python-daemon | grep runner /usr/share/pyshared/daemon/runner.py – Eared

I found that I needed to downgrade my version of lockfile for this to work (to version 0.8) – Backler

This suggestion seems to be obsolete -- as of September 2013, anyway, python.org/dev/peps/pep-3143 makes no mention of a "runner" that can be imported. This of course would explain @Nostradamnit's observation. – Bovid

This still works fine for me, in September 2013, on Ubuntu 13.04, with stock Python packages, python2.7 and python-daemon installed. With python3, however, I see error, " from daemon import runner ImportError: No module named 'daemon'" – Eared

@Bovid Probably you are not importing the module in the right way. Try from daemon import runner instead of import daemon – Defection

@Defection perhaps. See gist.github.com/d94a8d653d86f505e81a for what I did. – Bovid

if you've installed the 'daemon' package, it seems to take precedence of the 'python-daemon' package, and thus, your runner import will fail. – Alga

How do you verify if the daemon is running. The script runs without any error but when i run the top command, i cant see any python process. m on mac osx – Lovemaking

I am getting following error io.UnsupportedOperation: File or stream is not seekable. – Glair

Answer 4

An alternative -- create a normal, non-daemonized Python program then externally daemonize it using supervisord. This can save a lot of headaches, and is *nix- and language-portable.

Layman

I think this is the best way. Especially if you want to run several daemons on one operating system. Don't code, reuse. – Tusker

It simplifies a lot of issues. I've written true daemons -- they aren't easy. – Layman

The best answer is hidden here :) – Anticosti

This is gold. After spending hours trying to run through python-daemon, this is the out of the box solution that works for me. Great documentation and examples made my daemon up and running in few mins. – Lovemaking

Or, of you're using a modern Linux, run your program as a systemd-service... – Coatee

Answer 5

Note the python-daemon package which solves a lot of problems behind daemons out of the box.

Among other features it enables to (from Debian package description):

• Detach the process into its own process group.
• Set process environment appropriate for running inside a chroot.
• Renounce suid and sgid privileges.
• Close all open file descriptors.
• Change the working directory, uid, gid, and umask.
• Set appropriate signal handlers.
• Open new file descriptors for stdin, stdout, and stderr.
• Manage a specified PID lock file.
• Register cleanup functions for at-exit processing.

Evoke

Answer 6

Probably not a direct answer to the question, but systemd can be used to run your application as a daemon. Here is an example:

[Unit]
Description=Python daemon
After=syslog.target
After=network.target

[Service]
Type=simple
User=<run as user>
Group=<run as group group>
ExecStart=/usr/bin/python <python script home>/script.py

# Give the script some time to startup
TimeoutSec=300

[Install]
WantedBy=multi-user.target

I prefer this method because a lot of the work is done for you, and then your daemon script behaves similarly to the rest of your system.

Mireillemireles

This is the proper and sane way. 1) Needs to be saved to /etc/systemd/system/control.service 2) managed sudo systemctl start control.service – Zenia

Answer 7

YapDi is a python package. It can be used to convert a python script into daemon mode from inside the script.

Boyla

Answer 8

This function will transform an application to a daemon:

import sys
import os

def daemonize():
    try:
        pid = os.fork()
        if pid > 0:
            # exit first parent
            sys.exit(0)
    except OSError as err:
        sys.stderr.write('_Fork #1 failed: {0}\n'.format(err))
        sys.exit(1)
    # decouple from parent environment
    os.chdir('/')
    os.setsid()
    os.umask(0)
    # do second fork
    try:
        pid = os.fork()
        if pid > 0:
            # exit from second parent
            sys.exit(0)
    except OSError as err:
        sys.stderr.write('_Fork #2 failed: {0}\n'.format(err))
        sys.exit(1)
    # redirect standard file descriptors
    sys.stdout.flush()
    sys.stderr.flush()
    si = open(os.devnull, 'r')
    so = open(os.devnull, 'w')
    se = open(os.devnull, 'w')
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())

Agricola

Answer 9

since python-daemon has not yet supported python 3.x, and from what can be read on the mailing list, it may never will, i have written a new implementation of PEP 3143: pep3143daemon

pep3143daemon should support at least python 2.6, 2.7 and 3.x

It also contains a PidFile class.

The library only depends on the standard library and on the six module.

It can be used as a drop in replacement for python-daemon.

Here is the documentation.

Gerardgerardo

Answer 10

I am afraid the daemon module mentioned by @Dustin didn't work for me. Instead I installed python-daemon and used the following code:

# filename myDaemon.py
import sys
import daemon
sys.path.append('/home/ubuntu/samplemodule') # till __init__.py
from samplemodule import moduleclass 

with daemon.DaemonContext():
    moduleclass.do_running() # I have do_running() function and whatever I was doing in __main__() in module.py I copied in it.

Running is easy

> python myDaemon.py

just for completeness here is samplemodule directory content

>ls samplemodule
__init__.py __init__.pyc moduleclass.py

The content of moduleclass.py can be

class moduleclass():
    ...

def do_running():
    m = moduleclass()
    # do whatever daemon is required to do.

Preoccupancy

Answer 11

One more to thing to think about when daemonizing in python:

If your are using python logging and you want to continue using it after daemonizing, make sure to call close() on the handlers (particularly the file handlers).

If you don't do this the handler can still think it has files open, and your messages will simply disappear - in other words make sure the logger knows its files are closed!

This assumes when you daemonise you are closing ALL the open file descriptors indiscriminatingly - instead you could try closing all but the log files (but it's usually simpler to close all then reopen the ones you want).

Schott

Do you think opening a new logging handler is better than passing the logging handler through to the daemon using the DaemonContext's files_preserve option for example ? – Vincenza

You are only closing the logger, you aren't creating a new one (it'll just re-open it when it needs to). But even though it's really easy to do that, it might be better to use the DaemonContext as it's probably doing some other clever things (assuming preserving still allows proper daemonization). – Schott

Answer 12

Though you may prefer the pure Python solution provided by the python-daemon module, there is a daemon(3) function in libc -- at least, on BSD and Linux -- which will do the right thing.

Calling it from python is easy:

import ctypes

ctypes.CDLL(None).daemon(0, 0) # Read the man-page for the arguments' meanings

The only remaining thing to do is creation (and locking) of the PID-file. But that you can handle yourself...

Coatee

Answer 13

I modified a few lines in Sander Marechal's code sample (mentioned by @JeffBauer in the accepted answer) to add a quit() method that gets executed before the daemon is stopped. This is sometimes very useful.

Here it is.

Note: I don't use the "python-daemon" module because the documentation is still missing (see also many other SO questions) and is rather obscure (how to start/stop properly a daemon from command line with this module?)

Pogrom

Answer 14

After a few years and many attempts (I tried all the answers given here, but all of them had minor drawbacks at the end), now I realize that there is a better way than wanting to start, stop, restart a daemon directly from Python: use the OS tools instead.

For example, for Linux, instead of doing python myapp start and python myapp stop, I do this to start the app:

screen -S myapp python myapp.py    
# CTRL+A, D to detach

or screen -dmS myapp python myapp.py to start and detach it in one command.

Then:

screen -r myapp

to attach to this terminal again. Once in the terminal, it's possible to use CTRL+C to stop it.

Pogrom

This is useless if you want/need to run the daemon automatically, for example at system start. Also, it requires a terminal, which isn't quite what a "daemon" is all about. – Companionship

Answer 15

The easiest way to create daemon with Python is to use the Twisted event-driven framework. It handles all of the stuff necessary for daemonization for you. It uses the Reactor Pattern to handle concurrent requests.

Riptide

That's far too big a hammer to use. Most people just want to run a short Python script they wrote as a daemon. python-daemon, as described above, is the correct answer. – Fielder

Although this answer was quite arrogant, it was useful. – Newark

"easiest" and "Twisted" in one sentence… – Companionship

Answer 16

80% of the time, when folks say "daemon", they only want a server. Since the question is perfectly unclear on this point, it's hard to say what the possible domain of answers could be. Since a server is adequate, start there. If an actual "daemon" is actually needed (this is rare), read up on nohup as a way to daemonize a server.

Until such time as an actual daemon is actually required, just write a simple server.

Also look at the WSGI reference implementation.

Also look at the Simple HTTP Server.

"Are there any additional things that need to be considered? " Yes. About a million things. What protocol? How many requests? How long to service each request? How frequently will they arrive? Will you use a dedicated process? Threads? Subprocesses? Writing a daemon is a big job.

Hachure

Neither of those libraries even do a single fork(), let alone two. They have nothing to do with daemonization. – Consolute

How is a server not a daemon? Please explain. – Hachure

On Unix operating systems, a “daemon” process — like the aerial attendants that the Greeks called “daemons” — is one that “stands to the side.” Instead of directly serving a single user through that user's TTY, a daemon belongs to no TTY, but can answer requests from many users on the system, or — like crond or syslogd — does housekeeping services for the entire system. To create a daemon process, one must at least perform a double-fork() with all file descriptors closed, so that one is immune to signals from all controlling terminals, including the system console. See bignose's answer. – Consolute

@Brandon Craig Rhodes: I'm lost. I write a Python program that is a simple server process using SimpleHTTPServer. I use nohup to fork this Python server. How is that not a daemon? You've provided lots of words, but nothing that explains why a server is not a daemon. – Hachure

@S Lott — “a server” describes what a process does (listens for incoming requests instead of initiating its own actions); “a daemon” describes how a process runs (without a window or a controlling terminal). SimpleHTTPServer is indeed a server, but one that does not natively know how to daemonize itself (you can Ctrl-C it, for example). nohup is a utility to daemonize a naive process — so your nohupped server is indeed both a daemon and a server, exactly as you claim. This Stack Overflow question was essentially asking: “How can I implement nohup in Python?” – Consolute

@Brandon Craig Rhodes: "How can I implement nohup in Python?" Really? Amazing how you figured that out from almost no context clues. I'm impressed. I fail to see how that's possibly true, but since you keep posting comments, it must be true. I'm impressed. I've always found it needless to reimplement shell primitives in Python, but I guess that could be the right kind of answer. My preference is to provide an answer which avoids error-prone work. But. From your comments, the idea of avoiding errors is misplaced. Sorry. – Hachure

A "server" running as a user process in a terminal is not a "daemon" (e.g. a pydoc server that you start from the command line). A "daemon" that runs but doesn't serve anything to any clients (e.g. cron) is not a "server". The WSGI reference implementation at the SimpleHTTPServer do not daemonise and are not daemons. – Noelyn

@Noufal Ibrahim: And when I nohup SimpleHTTPServer, that doesn't show how to create a deamon with minimal work? I can't see why nohup both creates daemons and doesn't create daemons. – Hachure

Yes it does but my understanding of the OPs question is that he wants to do the daemonisation from within his python program and without using a something else. – Noelyn

@S Lott — You need not be impressed! The author of every other answer knew what “daemon” meant, so my ability to interpret this question is hardly unique. :) And where did you get the idea that I want the author to re-invent a wheel? I think nohup is a fine tool, and I will remove my -1 vote if you simply move that useful idea up into your actual answer. In fact, if you mention supervisord and how it will also save the author from having to do logging, a start-stop script, and restart throttling, then I'll even +1 you. :) – Consolute

After reading the following section from PEP 3143, entitled "A Daemon is Not a Service" (python.org/dev/peps/pep-3143/#a-daemon-is-not-a-service), I have concluded that S.Lott, though judgmental in their tone, is probably correct and that I don't want to use a daemon. – Consignor

indeed a server is normally wrapping a daemon, but it also performs additional head aching things that need filtering and handling, especially allows external access. so i'd witness myself i'd build a daemon until i might choose to consider direct external access. – Colloidal