Signal passing to managed processes using supervisord

Question

I am using supervisord to spawn and manage a FastCGI application that I am writing in C for a linux target. I have a signal handler that gracefully exits my application when SIGINT is received. I have verified that the signal handler works as desired by running the app in a terminal window and issuing Ctrl-C to exit.

When issuing a "shutdown" command to supervisord (via supervisorctl), it appears that supervisord is unable to force the app to exit without invoking SIGKILL:

2010-08-20 10:02:49,661 INFO waiting for cse to die
2010-08-20 10:02:52,665 INFO waiting for cse to die
2010-08-20 10:02:55,669 INFO waiting for cse to die
2010-08-20 10:02:58,672 INFO waiting for cse to die
2010-08-20 10:02:59,673 WARN killing 'cse' (2031) with SIGKILL
2010-08-20 10:02:59,674 INFO stopped: cse (terminated by SIGKILL)

I have the following in my supervisord.conf file

stopsignal=INT

It is my assumption that supervisord issues "stopsignal" at the invocation of the shutdown command, so I take the INFO statements as an indication that my app is not responding to the SIGINT issued by supervisord.

How do I go about debugging the signal passing between supervisord and my app?

Thwart

I confirmed that INT is issued at the invocation of the shutdown command, so the "...waiting for cse to die" log entries are definitely indicative of the application not seeing SIGINT. – Thwart

Answer 1

After some more digging, it appears that the issue is not that supervisord is failing to pass signals to child processes. This seems to be working normally.

Rather, it appears that supervisord stops logging child process stderr output once it receives a request to invoke stopsignal. As a result, any stderr-based logging of the child process shutdown will not be processed by supervisord. In my case, this made it appear that the child process was not receiving SIGINT, since it wasn't logging anything to stderr after the signal was invoked.

Thwart

Is there a way to get supervisord to continue collecting log output after a process has been sent SIGINT from a stop command? – Sixtasixteen

+1 on @Neil's question. Not seeing stdout and stderr makes impossible to debug problems during shutdown. – Gracie

Answer 2

You can run supervisord on the command line in debug mode and get more information.

supervisord -n -e debug

Endive

Thanks. The output that I originally posted was generated using the supervisord.conf equivalent of "-n -e debug" (don't daemonize and use debug level logging). – Thwart

Answer 3

This was a bug that has since been fixed in Supervisor's 3.0a10 (2011-03-30) release as per this commit: https://github.com/Supervisor/supervisor/commit/e19cbc185dfad045c8775750d36ab8ceed4c4dfb

Sixtasixteen