How to reload the mojolicious built-in webserver daemon?

Question

I copied this systemd service file from the lighttpd-package and adapted it to make systemd start the mojolicious built-in server daemon. Can I keep the ExecReload and the KillSignal setting for the mojolicious daemon?

[Unit]
Description=Start mojolicious.pl daemon
After=syslog.target network.target

[Service]
ExecStart=/path/to/mojolicious.pl daemon --listen "http://*:3001"
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Summer

Answer 1

I don't know why you would need "reload", Mojolicious performs internal reloads when application code changes. You might replace that to

/bin/kill -0 $MAINPID

if that line should be there, HUP would be terminal for normal application (unless you handle it differently)

Deli

Answer 2

I think Alex is thinking of morbo when he says it auto-reloads. For production you'd be better using hypnotoad. If you do switch to hypnotoad then there are various signals that will be useful. For example /bin/kill -USR2 $MAINPID will do a graceful reload, avoiding upset to any current connections. For more signals (for hypnotoad) see http://mojolicio.us/perldoc/Mojo/Server/Hypnotoad#SIGNALS

Fayre

Please read @RussellEGlaue's answer (that should be the accepted one in my opinion) before going forward with this. – Seismoscope

Answer 3

systemd reload and hypnotoad reload needs a wrapper script to work together properly

Note: This workaround applies to Mojolicious 6.x. SystemD compatibility was fixed in Mojolicious 7.x

If you are wanting to reload under systemd, then a standard hypnotoad reload (i.e. hypnotoad myapp.pl) or kill -USR $MAINPID is not preferrable. SystemD assumes that once the command to reload the hypnotoad server returns, the reload is finished. But, instead, both of these options exit immediately, and do not wait for the reload to finish.

When I have tried these standard reload options, systemd completes the execution of the reload command, and then, afterward, monitors that the pid file is removed and replaced when hypnotoad does the "Starting zero downtime software upgrade > upgrade successful".

If you have systemd configured to always run the hypnotoad app, systemd will then restart (stop and start) the application again - after hypnotoad did it gracefully.

Here is what happens:

• systemd reload hypnotoad-app (ExecReload=hypnotoad myapp.pl)
• hypnotoad-app exits after being told to reload
• hypnotoad starts up a new process with the newest myapp.pl
• systemd thinks the reload just completed, and sees that the pid did not change in the PIDFile
• hypnotoad see no load on the old processes, kills them, and deletes and creates a new PIDFile with the PID of the new process when the "Upgrade [is] successful" (This is hypnotoad's "zero downtime" reload procedure)
• systemd sees the pid file removed (it ignores that it is recreated, probably because by design, systemd wants to be in control of the forking)
• systemd waits for the HOLDOFF (RestartSec=<seconds>) if configured
• systemd does a full unit start of the hypnotoad-app

systemd documentation specifically says that when the ExecReload= command returns, systemd expects the process to already finished reloading, and the pid file to have the new PID of the reloaded process. But Hypnotoad does not work synchronously like that. So systemd and hypnotoad do not work together nicely.

https://www.freedesktop.org/software/systemd/man/systemd.service.html

Note however that reloading a daemon by sending a signal (as with the example line above) is usually not a good choice, because this is an asynchronous operation and hence not suitable to order reloads of multiple services against each other. It is strongly recommended to set ExecReload= to a command that not only triggers a configuration reload of the daemon, but also synchronously waits for it to complete.

The solution is to write a wrapper around the reload process.

#!/bin/bash

SERVER="/path/to/myapp.pl"
HYPNOTOAD="/usr/bin/hypnotoad"
PIDFILE="/var/run/myapp.pid"

# Timeout == LOOPSAFE x SLEEPTIME
SLEEPTIME="0.5"
LOOPSAFE=20
LOOPCOUNT=0
#

if [ ! -f "${PIDFILE}" ]; then
  # The PID files does not exist, maybe $SERVER is not running
  exit 1
fi

OLDPID=$(cat ${PIDFILE})
NEWPID=$OLDPID

# Reload the application
${HYPNOTOAD} ${SERVER}

while (( $LOOPCOUNT <= $LOOPSAFE )); do
  let LOOPCOUNT++
  if [ -f ${PIDFILE} ]; then
    NEWPID=$(cat ${PIDFILE})
    if (( $NEWPID > 1 )) && (( $NEWPID != $OLDPID )); then
      exit 0
    fi
  fi
  sleep ${SLEEPTIME}
done

exit 1

Here is the systemd unit file needed for Hypnotoad

[Unit]
Description=Hypnotoad-app
Requires=network.target
After=network.target

[Service]
Type=simple
SyslogIdentifier=hypnotoad-app
PIDFile=/var/run/myapp.pid
EnvironmentFile=-/etc/sysconfig/myapp
ExecStart=/usr/bin/hypnotoad --foreground /path/to/myapp.pl
ExecStop=/usr/bin/hypnotoad --stop /path/to/myapp.pl
ExecReload=/path/to/reload-myapp.sh
KillMode=process
Restart=always
RestartSec=5
User=myuser
Group=mygroup

[Install]
WantedBy=multi-user.target

Here is how the reload procedure now works

• systemd reload hypnotoad-app
• systemd executes the reload script
• reload script reloads hypnotoad-app, and waits for the "successful upgrade" - essentially sleeping until the PID file has the PID of the newly reloaded myapp.pl process
• systemd now monitors the pid file with the new PID, which does not change any more - so now systemd does not try to restart it

Liturgics

I submitted a bug/feature-request for a --reload-wait option to hypnotoad github.com/kraih/mojo/issues/1130 – Liturgics

Relaying the resolution of that issue back here. This issue was noticed around the same time that the documentation began to suggest systemd and it was resolved quickly. The poster of this answer was using an outdated version which did not include the fix (version 7.01 and above should work). – Lovelace

I was using Mojolicious 6.31 on CentOS 7. Upgrading Mojolicious to 7.45, and IO::Socket::IP to >= 0.37 (0.37+ is not available as an RPM for CentOS 7) works correctly with systemd. – Liturgics

Answer 4

Another option is to run hypnotoad in the foreground with -f & use service supervision with runit & a simple /etc/sv/myapp/run script:

#!/bin/sh

app=/path/to/my/app
daemon=/path/to/hypnotoad

exec 2>&1
exec $daemon -f $app

You can then have hypnotoad re-read it's config with a sv hup myapp

Red Hat doesn't make runit available but an rpm for the latest version can be built from here.

Don't forget to ln -s /etc/sv/myapp /etc/service/myapp to enable the service with runit.

Board