Ubuntu, upstart, and creating a pid for monitoring

Question

Below is a upstart script for redis. How to I create a pid so I use monit for monitoring?

#!upstart
description "Redis Server"

env USER=redis

start on startup
stop on shutdown

respawn

exec sudo -u $USER sh -c "/usr/local/bin/redis-server /etc/redis/redis.conf 2>&1 >> /var/log/redis/redis.log"

Tejada

Why would you want to use monit when you have already specified respawn? – Haun

@Haun The reason to use monit in addition to upstart/respawn is that respawn is aware only of whether the process is alive or not, but it does not know whether the application is in a bad state or not. Monit on the other hand can interact with the application in different ways, for example hitting an http status endpoint, to handle the scenarios where the process may be running but the app is in a bad or broken state that would indicate the process needs to be restarted. – Gifferd

Redis has it's own pid feature nowadays (pidfile in redis.conf) -- download.redis.io/redis-stable/redis.conf – Cartierbresson

Answer 1

If start-stop-daemon is available on your machine, I would highly recommend using it to launch your process. start-stop-daemon will handle launching the process as an unprivileged user without forking from sudo or su (recommended in the upstart cookbook) AND it also has built in support for pid file management. Eg:

/etc/init/app_name.conf

#!upstart
description "Redis Server"

env USER=redis

start on startup
stop on shutdown

respawn

exec start-stop-daemon --start --make-pidfile --pidfile /var/run/app_name.pid --chuid $USER --exec /usr/local/bin/redis-server /etc/redis/redis.conf >> /var/log/redis/redis.log 2>&1

Alternatively you could manually manage the pid file by using the post-start script stanza to create it and post-stop script stanza to delete it. Eg:

/etc/init/app_name.conf

#!upstart
description "Redis Server"

env USER=redis

start on startup
stop on shutdown

respawn

exec sudo -u $USER sh -c "/usr/local/bin/redis-server /etc/redis/redis.conf 2>&1 >> /var/log/redis/redis.log"

post-start script
    PID=`status app_name | egrep -oi '([0-9]+)$' | head -n1`
    echo $PID > /var/run/app_name.pid
end script

post-stop script
    rm -f /var/run/app_name.pid
end script

Gifferd

Am I missing something, or will the start-stop-daemon version never work correctly? Wouldn't Upstart track the PID of start-stop-daemon and assume the job had terminated when start-stop-daemon returned? This is the behavior I'm seeing in testing, and as far as I can tell, it's by design. – Evangelia

start-stop-daemon works like a transparent middle-man between upstart and the process specified in the --exec parameter of start-stop-daemon. All your upstart commands and PID monitoring should behave as if they were attached directly to the process in --exec. Could you provide some more information as to the kind of testing that seems to be producing different results? – Gifferd

Do you have also to chown $USER to the pid file? – Criticaster

@JoséF.Romaniello No need, because upstart and monit both run as root and therefore should be able to access the pid file. Giving an unprivileged user access to the pid file sounds like a security concern and if you have that use case you may want to find a different way to solve the problem. – Gifferd

Is there still a reason to use start-stop-daemon since upstart scripts support 'setuid' now? Not sure I understand the benefit of s-s-d. – Shower

@Shower According to the upstart cookbook (linked in my reply) the setuid and setgid stanzas are only available in upstart 1.4+. Also they say for setuid: "Note that all processes (pre-start, post-stop, et cetera) will be run as the user specified." whereas s-s-d could target only a single process within your upstart script to be run as the nonprivileged user (edge case I think, but maybe someone might need this). setuid and s-s-d both solve the problem of forking from sudo or su, but most importantly for the OP's question, s-s-d provides PID management in addition. – Gifferd

Interestingly, why is the author of upstart recommending not to mix start-stop-daemon and upstart? bugs.launchpad.net/ubuntu/+source/upstart/+bug/485721 – Newmark

@Newmark Wow you are right... I wonder why he would say this? It seems odd that the upstart cookbook would contain such a direct contradiction. I really don't know what to make of it. Perhaps the author was only referring to the way the user who posted the bug mixed them inside of the script directives, and maybe it is okay to use it in an exec directive? I could only guess what the disconnect is but nice find. – Gifferd

can i do this with the user id starting as root? The daemon i'm launching already delevates itself properly – Baucis

@Baucis Yes it should be easy, in the start-stop-daemon flags, just specify the --chroot flag (without an argument) instead of --chuid. Actually since the start-stop-daemon command is being run as root to begin with you probably don't even need the --chroot flag at all. – Gifferd

Answer 2

Egg's 1st example with start-stop-daemon is way to go.

If You choose 2nd, I would suggest $$ to obtain the PID.

#!upstart
description "Redis Server"

env USER=redis

start on startup
stop on shutdown

respawn

script
    echo $$ > /var/run/app_name.pid
    exec sudo -u $USER sh -c "/usr/local/bin/redis-server /etc/redis/redis.conf 2>&1 >> /var/log/redis/redis.log"
end script

post-stop script
    rm -f /var/run/app_name.pid
end script

Sergu

this was the only solution that worked for my logstash problems. Also note that the script initially failed because setuid requires sudo apt-get install super. Must be an apt broken dependency in the bundled upstart script. – Shing