A better way to restart/reload Gunicorn (via Upstart) after 'git pull'ing my Django projects

Question

Im looking for something better than sudo restart projectname every time I issue a git pull origin master, which pulls down my latest changes to a Django project. This restart command, I believe, is related to Upstart, which I use to start/top my Gunicorn server process.

This restart causes a brief outage. Users hitting the web server (nginx) will get a 500, because Gunicorn is still restarting. In fact, it seems to restart instantly, but it takes a few seconds for pages to load.

Any ideas on how to make this seamless? Ideally, I'd like to issue my git pull and Gunicorn reloads automatically.

Lingerie

Answer 1

For a graceful reload, you should instead use Upstart's reload command, e.g.:

sudo reload jobname

According to the initctl (Upstart) manpage, reload will send a HUP signal to the process:

reload JOB [KEY=VALUE]...

       Sends the SIGHUP signal to running process of the named JOB instance.

...which for Gunicorn will trigger a graceful restart (see FAQ).

Makell

noob question: and is jobname something I get from the process itself or an option I passed into gunicorn? – Needful

@TheGrimmScientist, jobname is the name of your upstart job/config. E.g. For the upstart job at /etc/init/foo.conf, foo would be the jobname. – Makell

@Makell Is there any example available? – Forgo

@Dewsworld, an example upstart config? Many examples can be found, e.g. #17748105 For a full reference, see: upstart.ubuntu.com/cookbook – Makell

In the context of this question, try sudo reload /etc/init/gunicorn.conf. This assumes you've got gunicorn.conf at /etc/init/. – Weeks

Answer 2

You can tell Gunicorn to reload gracefully using the HUP signal like so:

kill -HUP <pid>

(see the FAQ for details)

I use Supervisor to control my Gunicorn server, which allows me to use this (slightly hacky) way of reloading Gunicorn after a deploy:

supervisorctl status gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP

You could obviously achieve something similar with pidof, or ps.

This is actually run from a Fabric script, so I don't even have to logon to the server at all.

Leandra

with supervisor you can still ask gunicorn to write its pid to a file if you want with pidfile but BE CAREFUL to put it somewhere where the user setup by supervisor to run gunicorn will have the right to write! – Despotic

Thanks - that's a neater way of doing it! – Leandra

actually, the pidfile setting is only supported for the supervisord process itself. And instead of all that sed'ing, why not use supervisorctl pid gunicorn | xargs ... ?! – Alberta

pid = Get the PID of supervisord not of the subprocess – Drypoint

From supervisorctl help pid: pid <name> Get the PID of a single child process by name. (Which a quick test verifies). Without the name it's supervisorctl's PID. – Tattan

@Tattan I disagree. I've just tested it $ sudo supervisorctl pid program1, $ sudo supervisorctl pid program2, $ sudo supervisorctl pid program3, and I get every time the same number. Perhaps it has changed in newer versions (I'm using 3.0a8) – Damron

At least on supervisor 3.0, sudo supervisorctl pid gunicorn does indeed return the main gunicorn (not supervisord) pid. So then you can run sudo kill -HUP `sudo supervisorctl pid gunicorn` to gracefully restart Gunicorn. However note that this doesn't necessarily mean zero issues during deployment, because the old workers stop taking requests and your new workers Python app takes some time to boot up, you'll get hanging requests while the new workers get started. – Lashawnda

How does -HUP differ from supervisorctl reload? – Grubbs

@ScottStafford, supervisorctl reload asks the supervisor daemon to reload its config. Perhaps you meant supervisorctl restart? This is just shorthand for stop followed by start, so the gunicorn master and all workers are clobbered. Using SIGHUP leaves the gunicorn master running; new workers are created, old ones gracefully killed. No requests are dropped. Much cleaner. – Cleanse

just an easier version when using supervisor: supervisorctl pid NAME | xargs kill -HUP – Disclamation

supervisorctl signal HUP NAME should also do the trick. supervisord.org/running.html#signals – Phenomenology

Answer 3

For those not using supervisord: what Rob said, it works with ps as well,

ps aux |grep gunicorn |grep projectname | awk '{ print $2 }' |xargs kill -HUP

Cubic

What would be the difference between these: $ ps -A | grep gunicorn | awk '{print $1}' | xargs \\\ $ "15135 15189 15193 15196" \\\ $ ps aux | grep gunicorn | grep staging | awk '{print $2}' \\\ $ "15193 15196" \\\ obviously, your version is not picking up all of the gunicorn processes. Only resetting your version worked fine. But what are these other two processes? – Diarist

Answer 4

Systemd, gunicorn & Ubuntu

Here is the one-liner, if you are running your gunicorn service with systemd.

systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

---

Details step by step

Since the gunicorn docs tell that the correct way to gracefully reload the workers is by using kill -HUP <Main PID>, where <Main PID> is the process id of the master process, we extract the master PID using systemctl, and run kill -HUP <Main PID>.

1) Get info about the process from systemd using the name of the service

systemctl status gunicorn 

where gunicorn is the name of the service, located at /etc/systemd/system/.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn
● gunicorn.service - Gunicorn server for yourproject.com
   Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-11-04 19:16:24 UTC; 1h 15min ago
 Main PID: 10673 (gunicorn)
   CGroup: /system.slice/gunicorn.service
           ├─10673 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11069 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11070 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           └─11071 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application

Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11047] [INFO] Booting worker with pid: 11047
Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11048] [INFO] Booting worker with pid: 11048
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Handling signal: hup
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Hang up: Master
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11046] [INFO] Worker exiting (pid: 11046)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11047] [INFO] Worker exiting (pid: 11047)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11048] [INFO] Worker exiting (pid: 11048)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11069] [INFO] Booting worker with pid: 11069
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11070] [INFO] Booting worker with pid: 11070
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11071] [INFO] Booting worker with pid: 11071

2) Get the process ID (PID) of the main gunicorn process

The sed command works like follows: sed 's/<search this>/<replace with this>/g'

• s means for the substitute command, and g means that search the whole input globally.
• The -n flag tells sed not to print every line (or actually, not to print anything.)
• The p at the end tells sed to print the matched line.
• We search for .*Main PID: \(.*\)$, which is a regular expression pattern, which has following parts: .* matches any character (.) zero or more times (*). Then we search for Main PID: followed by any characters, repeated zero or more times (.*). To capture all characters after the Main PID:-text, we enclose the .* into parenthesis, which are escaped with backslashes: \(.*\). $ indicates line end.
• The "replace with this" part of the sed command is just \1, which means the first captured set of characters.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p'
10673 (gunicorn)

3) Get rid of the extra characters

Pipe the output to cut. The cut -f1 -d' ' means, that

• The string is space delimited: Here -d determines the delimiter, which is the characted just after the -d. Since the delimiter is space, we enclose that in quotes.
• -f means just that cutting is made using the delimiter (and not by bytes), and -f1 means that we want to take out the first element of the list.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' '
10673

4) Use the Main PID

Piping to xargs means just running the command with arguments from the pipe on the left hand side. Since we are piping just the Main PID to xargs,

 systemctl status gunicorn-django |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

is basically just the same thing as

echo <Main PID > | xargs kill -HUP

which translates into

kill -HUP <Main PID >

---

Edit

A little more robust solution would be to use cut -f1 -d$'\n' or grep -m1 "" in front of cut -f1 -d' ', to pick just the first line of the match. I can't figure out any circumstances, where there would be two matches for the Main PID:, though.

Resignation

Unit gunicorn.service could not be found. while I have installed gunicorn via pip. – Comprehend

Uhm, you can just ask systemd nicely for the PID: systemctl show gunicorn -p MainPID --value. A One-liner would be: kill -HUP $(systemctl show gunicorn -p MainPID --value) – Chirk

Answer 5

Maybe not a direct answer to the question, but for those who are just looking for a way to restart gunicorn web server, you can use killall gunicorn and then execute a command to start gunicorn again. For example:

killall gunicorn
gunicorn --bind 0.0.0.0:80 --reload app:app

Note: killall gunicorn will terminate all gunicorn processes immediately so make sure you understand what you are doing.

Locker

While that will work, it’s likely bad practice. Reloading gunicorn gracefully will allow active connections to finish as they should. Just killing things will break those connections and if something is left open And gunicorn ain’t there to say close it, you could get into some lame issues. However, If your site/project/constraints are simple enough this is fine to me, sure. – Lingerie

killall can send HUP as well. Just run killall -s HUP gunicorn. – Chirk

Answer 6

We run Gunicorn under Supervisor, but this is the simplest, cleanest way we've found to gracefully reload Gunicorn when it gets confused:

sudo pkill -HUP -f gunicorn.*master

Pratt

Answer 7

sudo systemctl restart gunicorn

Materialize

Unfortunately this will also give users hitting the web server (nginx) a 500, because Gunicorn is still restarting – Centroid

Answer 8

If you are running gunicorn on a port rather than a socket, you can find the process id (pid) of gunicorn using fuser command. Then force gunicorn to reload the code by sending a HUP signal.

The command fuser 8000/tcp will list the process ids of all processes using tcp port 8000.

fuser -k 8000/tcp will kill those processes gracelessly which is not recommended.

• fuser -k -HUP 8000/tcp will force gunicorn using tcp port 8000 to reload the code by sending HUP signal.

Getaway

Answer 9

When using systemd

calling:

systemctl restart gunicorn

resulted in 502 bad gateway errors for us. Digging into the nginx logs we saw that the socket file was not found. Calling

systemctl restart gunicorn.socket

after restarting gunicorn, restarted the socket and brought the server back online.

Worldly

Answer 10

You can start gunicorn with the code:

gunicorn  --pid /var/run/gunicorn.pid --bind 0.0.0.0:80 --reload myproject.asgi.application

And graceful restart with the code:

cat /var/run/gunicorn.pid | xargs kill -HUP

Buttonhole

Answer 11

Modern and easier way to handle the reload with systemd:

Add the ExecReload= directive to the unit file:

[Service]
ExecReload=kill -HUP $MAINPID

Now you can simply use sudo systemctl reload unit.

Chirk

Answer 12

Restart gunicorn service using cmd
systemctl restart gunicorn

else restart gunicorn service and create the sock file again.

Ulcerative

Answer 13

This can help when gunicorn restart(in a dev/test env) takes a lot of time to restart.

The below command kills the workers.

$ sudo kill -9 `sudo lsof -n -i | grep gunicorn | awk '{print $2}'`

$ sudo service gunicorn restart

Subclavius