python sigkill catching strategies

Question

I was wondering if there was any way to catch the sigkill from the OOM killer. I have a task queue, and every so often a mammoth task is created that is killed by OOM. This:

catch Exception as ex:
    # clean up!

does not work, as SIGKILL can't be caught. So........is there ANY strategy to clean up after a SIGKILL? Can I fork, and watch the child process? If so, any resources opened by the child process would have to be known in advance by the parent? Or could I just do some version of

ps -ef | grep <child pid> | xargs kill -9  (you get the idea...)

Currently, if I don't clean up after an OOM kill, I leave behind plenty of child processes and other things that just make it worse when the task is retried, and soon enough, the server is unreachable.

Finally, is it enough to just do:

kill -9 <process id> 

to test this exact situation?

Thanks very much!

Jacquesjacquet

Answer 1

SIGKILL by its very nature cannot be trapped.

See http://en.wikipedia.org/wiki/Unix_signal#SIGKILL:

SIGKILL

The SIGKILL signal is sent to a process to cause it to terminate immediately (kill). In contrast to SIGTERM and SIGINT, this signal cannot be caught or ignored, and the receiving process cannot perform any clean-up upon receiving this signal.

The best thing to do is the next time your process launches, look for anything that needs to be cleaned up.

And yes, kill -9 <pid> will send a SIGKILL to the process. (To be precise, it sends the 9th signal - it just happens that SIGKILL has the number 9 on pretty much every system. You could alternatively write kill -KILL <pid>, which lets you specify the signal by name instead of by number in a portable way.)

Coelenteron

Is there any best practice or conventional wisdom on HOW to clean up? If i have 10 things opening a subprocess...should I look for process with parent id == 1, and force kill them? I'm a little surprised there isn't a common way to do this...but I guess if it's (supposed to be) a pretty uncommon thing, everyone comes up with their own? – Jacquesjacquet

If you have a bunch of subprocesses, they would have to have some mechanism to monitor the other processes if they get killed. – Coelenteron

FWIW, I used the excellent psutil library to do exactly this - clean up on the next run of the process, by searching for expected process names with the ppid of 1, owned by the process user. – Jacquesjacquet

Answer 2

The Linux OOM killer works by sending SIGKILL.

To kill the selected process, the OOM killer delivers a SIGKILL signal.

kill -9 <-- Works

Tisdale