Can upstart expect/respawn be used on processes that fork more than twice?

Question

I am using upstart to start/stop/automatically restart daemons. One of the daemons forks 4 times. The upstart cookbook states that it only supports forking twice. Is there a workaround?

How it fails

If I try to use expect daemon or expect fork, upstart uses the pid of the second fork. When I try to stop the job, nobody responds to upstarts SIGKILL signal and it hangs until you exhaust the pid space and loop back around. It gets worse if you add respawn. Upstart thinks the job died and immediately starts another one.

Bug acknowledged by upstream

A bug has been entered for upstart. The solutions presented are stick with the old sysvinit, rewrite your daemon, or wait for a re-write. RHEL is close to 2 years behind the latest upstart package, so by the time the rewrite is released and we get updated the wait will probably be 4 years. The daemon is written by a subcontractor of a subcontractor of a contractor so it will not be fixed any time soon either.

Strobile

Answer 1

I came up with an ugly hack to make this work. It works for my application on my system. YMMV.

1. start the application in the pre-start section
2. in the script section run a script that runs as long as the application runs. The pid of this script is what upstart will track.
3. in the post-stop section kill the application

example

env DAEMON=/usr/bin/forky-application

pre-start script
    su -s /bin/sh -c "$DAEMON" joeuseraccount
end script

script 
    sleepWhileAppIsUp(){
        while pidof $1 >/dev/null; do
            sleep 1
        done
    }

    sleepWhileAppIsUp $DAEMON
end script

post-stop script
    if pidof $DAEMON;
    then
        kill `pidof $DAEMON`
        #pkill  $DAEMON # post-stop process (19300) terminated with status 1
    fi
end script

a similar approach could be taken with pid files.

Strobile

My service command is stuck. If this is the solution in the long term. How do I refresh my service command (to forget about the service that was forking too many times) short of restarting the server? – Lawrencelawrencium

@Strobile I'm currently using my script. It starts perfectly, but after it has started and I write status. It gives me: supervisord stop/waiting and it's not possible to use service supervisord stop. I thought it may have something to do with bugs.launchpad.net/upstart/+bug/406397/comments/47 but I'm not sure what he means. – Lawrencelawrencium

I fixed it, I basically replaced all mentions of pidof with pgrep. Seems like pidof wasn't working with me. – Lawrencelawrencium

@CMCDragonka if you do initctl log-priority debug you can see the jobs/services change status in /var/log/messages. If upstart is waiting on a pid that doesnt exist anymore then you need to exhaust your pid space so a new program pops up with the old pid. upstart will then kill that process and go back to a sane state. Creating a bunch of sleeps works well for this. – Strobile

I found I had to also add an initctl emit <service-name> & to the pre-start script, to get any dependent services to notice when this one comes up. The & seems to be required to stop it from hanging when re-start ing after a stop. restart doesn't work. overall, it's serviceable, but further tips+insights appreciated? – Unaccountable

PS - my use case was to get sshfs to start on boot. Just mentioning this because I've seen others struggling with it too, so hopefully google searches will now find this page – Unaccountable

In order to make this work with respawn (so the service keeps running even if it was killed or has crashed), all it took was adding an exit with non-zero code at the end of the main script section. This tells Upstart this was an abnormal exit and the daemon needs respawning. For this to work it's important that the post-stop script finishes cleanly; so indeed a brute-force pkill won't do, as suggested by comments. – Witter