Daemon vs Upstart for python script

Question

I have written a module in Python and want it to run continuously once started and need to stop it when I need to update other modules. I will likely be using monit to restart it, if module has crashed or is otherwise not running.

I was going through different techniques like Daemon, Upstart and many others.

Which is the best way to go so that I use that approach through out my all new modules to keep running them forever?

Acetone

Answer 1

From your mention of Upstart I will assume that this question is for a service being run on an Ubuntu server.

On an Ubuntu server an upstart job is really the simplest and most convenient option for creating an always on service that starts up at the right time and can be stopped or reloaded with familiar commands.

To create an upstart service you need to add a single file to /etc/init. Called <service-name>.conf. An example script looks like this:

description "My chat server"
author "[email protected]"

start on runlevel [2345]
stop on runlevel [!2345]

env AN_ENVIRONMENTAL_VARIABLE=i-want-to-set

respawn

exec /srv/applications/chat.py

This means that everytime the machine is started it will start the chat.py program. If it dies for whatever reason it will restart it. You don't have to worry about double forking or otherwise daemonizing your code. That's handled for you by upstart.

If you want to stop or start your process you can do so with

service chat start 
service chat stop

The name chat is automatically found from the name of the .conf file inside /etc/init

I'm only covering the basics of upstart here. There are lots of other features to make it even more useful. All available by running man upstart.

This method is much more convenient, than writing your own daemonization code. A 4-8 line config file for a built in Ubuntu component is much less error prone than making your code safely double fork and then having another process monitor it to make sure it doesn't go away.

Monit is a bit of a red herring. If you want downtime alerts you will need to run a monitoring program on a separate server anyway. Rely on upstart to keep the process always running on a server. Then have a different service that makes sure the server is actually running. Downtime happens for many different reasons. A process running on the same server will tell you precisely nothing if the server itself goes down. You need a separate machine (or a third party provider like pingdom) to alert you about that condition.

Cymatium

Is there any authentic source that supports your answer? Running on Ubuntu is an option but not compulsory (daemonize will work on both). Second you have to use monit even with upstart to get downtime alerts. – Acetone

You can get upstart to email you when it stops/starts (serverfault.com/questions/236925/…). authentic source? People use upstart in production everywhere. We use upstart in production You can't use monit for downtime alerts. It's pointless. See edited answer for the reason why. – Cymatium

Great post. Upstart is in the Debian tree, so it probably can be used in vanilla debian or Mint as well as other derivatives. – Wickham

Just crossreferencing the following post, since this is one of the answers I used in solving this for myself. Basically you can create a proper daemon easily with python-daemon, and then run it as a service as above, with a few gotchas: https://mcmap.net/q/261315/-python-script-as-linux-service-daemon – Interpenetrate

Basically you need to carefully figure out what happens after you run the command, does it fork (create a new subprocess)? If it does fork. Does it fork once or twice? The upstart cookbook explains this very well: upstart.ubuntu.com/cookbook/#expect. – Cymatium

If you are simply writing a little Python service there is no reason to use python-daemon AND upstart. Just let upstart take care of restarting your script if it fails. That sounds like it was the cause of your problem Ross. – Cymatium

Then again, if upstart is just one of the init implementations the daemon might run on, the daemon library is still extremely useful.You just need a way to set detach_process kwarg to False, when the specific init manages your daemon. This also applies to the old-school inetd for example. The python-daemon still provides you with a clean context which you can clean up on exiting. – Grume

If you're wanting to package your software across platforms I'd argue you would want to really package it for each distribution :) Which now-a-days would generally involve writing a systemd unit and install it properly. Working with the host operating system in a standard way has benefits. – Cymatium

Answer 2

You could check out supervisor. What it is capable of is starting a process at system startup, and then keeping it alive until shutdown.

The simplest configuration file would be:

[program:my_script]
command = /home/foo/bar/venv/bin/python /home/foo/bar/scripts/my_script.py
environment = MY_ENV_VAR=FOO, MY_OTHER_ENV_VAR=BAR
autostart = True
autorestart = True

Then you could link it to /etc/supervisord/conf.d, run sudo supervisorctl to enter management console of supervisor, type in reread so that supervisor notices new config entry and update to display new programs on the status list.

To start/restart/stop a program you could execute sudo supervisorctl start/restart/stop my_script.

Rumen

A general note to readers that supervisor does not work with Python 3. – Whenas

Answer 3

I used old-style initscript with start-stop-daemon utility.Look at skel in /etc/init.d

Vulnerary