supervisord stopping child processes

Question

One of the problems, I face with supervisord is that when I have a command which in turn spawns another process, supervisord is not able to kill it.

For example I have a java process which when runs normally is like

 $ zkServer.sh start-foreground
 $ ps -eaf | grep zk
 user 30404 28280  0 09:21 pts/2    00:00:00 bash zkServer.sh start-foreground
 user 30413 30404 76 09:21 pts/2    00:00:10 java -Dzookeeper.something..something

The supervisord config file looks like:

[program:zookeeper]
command=zkServer.sh start-foreground
autorestart=true
stopsignal=KILL

These kind of processes which have multiple childs are not well handled by supervisord when it comes to stopping them from supervisorctl. So when I run this from the supervisord and try to stop it from supervisorctl, only the top level process gets killed but not the actual java process.

Pamilapammi

It is my understanding that the systemd init-replacement's use of cgroups allows for reliably tracking child processes. It might be suitable for your needs. – Patrizia

Answer 1

The same problem was encountered by Rick Hanlon II here: https://coderwall.com/p/4tcw7w

Option stopasgroup=true should be set in the program section for supervisord to stop not only the parent process but also the child processes.

The example is given as:

[program:some_django]
 command=python manage.py runserver
 directory=/dir/to/app
 stopasgroup=true

Also, have in mind that you may have an older package of supervisord that does not have "stopasgroup" functionality. I tried these Debian packages on Raspberry Pi:

• supervisor_3.0a8 does not work.
• supervisor_3.0b2-1 works as expected.

Becnel

Also note that supervisord doesn't take configuration changes automatically. You'll need to run supervisorctl update to apply changes to your configuration, or restart the supervisord process. – Goggler

This should really be the default option. – Davie

This worked perfectly for me with supervisor 3.2. Thanks. – Gag

Answer 2

Doing the following early in the main bash script called by supervisord fixed the problem for me:

trap "kill -- -$$" EXIT

This kills the entire process group when the main script exits, such as when it is killed by supervisord.

Gunshy

Answer 3

try this supervisor program config:

stopasgroup=true
killasgroup=true
stopsignal=INT

Capacitate

Note: stopasgroup=true implies killasgroup=true. no need to mention again. supervisord.org/configuration.html – Vaseline

Answer 4

A feature was recently added to supervisord to send SIGKILL to the whole process group. It's in github but not officially released yet.

If the process id is available in a file, you can use the pid-proxy program

Largely

Answer 5

The following article has an in-depth discussion of the problem:

http://veithen.github.io/2014/11/16/sigterm-propagation.html

Remy

exec ... solved it for me. The first option in the article. Thanks – Absinthe

Thank you for teaching how to catch the fish instead of supplying it ;) – Protostele

Answer 6

You can also use priorities in /conf.d/your-configuration.conf file. For example, if you want to run zookeeper first and then kafka you can specify two programs.

Lower priority means that the program starts first and stops last.

Porbeagle