Killing the children with the parent

Question

I have a program spawning and communicating with CPU heavy, unstable processes, not created by me. If my app crashes or is killed by SIGKILL, I want the subprocesses to get killed as well, so the user don´t have to track them down and kill them manually.

I know this topic has been covered before, but I have tried all methods described, and none of them seem to live up to survive the test.

I know it must be possible, since terminals do it all the time. If I run something in a terminal, and kill the terminal, the stuff always dies.

I have tried atexit, double fork and ptys. atexit doesn't work for sigkill; double fork doesn't work at all; and ptys I have found no way to work with using python.

Today, I found out about prctl(PR_SET_PDEATHSIG, SIGKILL), which should be a way for child processes to order a kill on themselves, when their parent dies. I tried to use it with popen, but it seams to have no effect at all:

import ctypes, subprocess
libc = ctypes.CDLL('/lib/libc.so.6')
PR_SET_PDEATHSIG = 1; TERM = 15
implant_bomb = lambda: libc.prctl(PR_SET_PDEATHSIG, TERM)
subprocess.Popen(['gnuchess'], preexec_fn=implant_bomb)

In the above, the child is created and the parent exits. Now you would expect gnuchess to receive a SIGKILL and die, but it doesn't. I can still find it in my process manager using 100% CPU.

Can anybody tell me if there is something wrong with my use of prctl?, or do you know how terminals manage to kill their children?

Kobarid

I find this title very disturbing :( – Ghoul

Actually, I found that your approach seemed to work just fine - perhaps it's just that your test program (gnuchess) actually does some double-forking, or something similar? I tested with just a simple python script - see my "answer" below for the exact test code I used. (Sorry, would have included the code in the comment, but I guess you can't have code blocks in them...) – Skiver

Answer 1

prctl's PR_SET_DEATHSIG can only be set for this very process that's calling prctl -- not for any other process, including this specific process's children. The way the man page I'm pointing to expresses this is "This value is cleared upon a fork()" -- fork, of course, is the way other processes are spawned (in Linux and any other Unix-y OS).

If you have no control over the code you want to run in subprocesses (as would be the case, essentially, for your gnuchess example), I suggest you first spawn a separate small "monitor" process with the role of keeping track of all of its siblings (your parent process can let the monitor know about those siblings' pids as it spawns them) and sending them killer signals when the common parent dies (the monitor needs to poll for that, waking up every N seconds for some N of your choice to check if the parent's still alive; use select to wait for more info from the parent with a timeout of N seconds, within a loop).

Not trivial, but then such system tasks often aren't. Terminals do it differently (via the concept of a "controlling terminal" for a process group) but of course it's trivial for any child to block THAT off (double forks, nohup, and so on).

Habile

preexec_fn is called after fork(), and the man page doesn't say this flag is cleared on exec(). – Bargain

As Denis writes, I'm under the impression, that the preexec_fn parameter will place my prctl call between the fork and the exec calls. The idea of a monitor is actually quite good. Unless it crashes of course, so it has to be quite simple. Can I know its parrent is dead when it recieves SIGHUP? And then let it sigterm its brothers and sisters. Can you tell me more about spawning controlling terminals? – Kobarid

From [Python]: Popen Constructor (probably changed along the way): "If preexec_fn is set to a callable object, this object will be called in the child process just before the child is executed. (POSIX only)." – Tempest

Answer 2

I know it's been years, but I found a simple (slightly hacky) solution to this problem. From your parent process, wrapping all your calls around a very simple C program that calls prctl() and then exec() solves this problem on Linux. I call it "yeshup":

#include <linux/prctl.h>
#include <signal.h>
#include <unistd.h>

int main(int argc, char **argv) {
     if(argc < 2)
          return 1;
     prctl(PR_SET_PDEATHSIG, SIGHUP, 0, 0, 0);
     return execvp(argv[1], &argv[1]);
}

When spawning your child processes from Python (or any other language), you can run "yeshup gnuchess [argments]." You'll find that, when the parent process is killed, all your child processes (should) be given SIGHUP nicely.

This works because Linux will honor the call to prctl (not clear it) even after execvp is called (which effectively "transforms" the yeshup process into a gnuchess process, or whatever command you specify there), unlike fork().

Unaesthetic

Can some variant of this fly for a bash-based test case? I've been monkeying with it without any luck. – Uprise

But how is this different from calling fork(); prctl(); exec()? – Kobarid

Thomas: That might work, but remember, OP is trying to call a subprocess from Python. – Unaesthetic

@Unaesthetic I know, but the preexecfn in Python is exactly(?) like plugging a call I'm between fork and exec. Which is equivalent to this solution. – Kobarid

@Unaesthetic - i think this is the best solution - also thoughts on adding a if (getppid() == 1) return 1; as a check after prctl() - that way if the parent process dies and init takes over the orphan as a parent before prctl() , you're not waiting on init to die (you could be waiting a while :) – Skull

Answer 3

Actually I found that your original approach worked just fine for me - here's the exact example code I tested with which worked:

echoer.py

#!/bin/env python

import time
import sys
i = 0
try:
    while True:
        i += 1
        print i
        time.sleep(1)
except KeyboardInterrupt:
    print "\nechoer caught KeyboardInterrupt"
    exit(0)

parentProc.py

#!/bin/env python

import ctypes
import subprocess
import time

libc = ctypes.CDLL('/lib64/libc.so.6')
PR_SET_PDEATHSIG = 1
SIGINT = 2
SIGTERM = 15

def set_death_signal(signal):
    libc.prctl(PR_SET_PDEATHSIG, signal)

def set_death_signal_int():
    set_death_signal(SIGINT)

def set_death_signal_term():
    set_death_signal(SIGTERM)

#subprocess.Popen(['./echoer.py'], preexec_fn=set_death_signal_term)
subprocess.Popen(['./echoer.py'], preexec_fn=set_death_signal_int)
time.sleep(1.5)
print "parentProc exiting..."

Skiver

doesn't work "better", just different - to see the diff for yourself (on a linux system), create both the above files in the same dir, make them executable, and then run "parentProc.py". The child process should get a KeyboardInterrupt signal sent to it, which you'll know because it will print "echoer caught KeyboardInterrupt". If you change parentProc.py to use set_death_signal_term, the echoer will still get killed, but in a more abrupt fashion. – Skiver

Answer 4

I thought the double fork was to detach from a controlling terminal. I'm not sure how you are trying to use it.

It's a hack, but you could always call 'ps' and search for the process name your trying to kill.

Ultraviolet

But how can I call ps when my controlling process is dead? That's the entire point. The user of my app won't know that the children aren't dead, and will just feel his/her computer being much slower until reboot. – Kobarid

Can you wrap the program that is being created with your own executable and then write the PID to a file when its started? – Ultraviolet

I could probably find a way to clean stuff up on restart, but I don't know if I will be restarted. Thus I need a way to clean up at the time I die. – Kobarid

Answer 5

I've seen very nasty ways of "clean-up" using things like ps xuawww | grep myApp | awk '{ print $1}' | xargs -n1 kill -9

The client process, if popened, can catch SIG_PIPE and die. There are many ways to go about this, but it really depends on a lot of factors. If you throw some ping code (ping to parent) in the child, you can ensure that a SIG_PIPE is issued on death. If it catches it, which it should, it'll terminate. You'd need bidirectional communication for this to work correctly... or to always block against the client as the originator of communication. If you don't want to modify the child, ignore this.

Assuming that you don't expect the actual Python interpreter to segfault, you could add each PID to a sequence, and then kill on exit. This should be safe for exiting and even uncaught exceptions. Python has facilities to perform exit code... for clean-up.

Here's some safer nasty: Append each child PID to a file, including your master process (separate file). Use file locking. Build a watchdog daemon that looks at the flock() state of your master pid. If it's not locked, kill every PID in your child PID list. Run the same code on startup.

More nasty: Write the PIDs to files, as above, then invoke your app in a sub-shell: (./myMaster; ./killMyChildren)

Bell

If you read my codesnap, you'll see that I run gnuchess in the new process, so there is now way I can run it as a thread. Does Popen not use fork? – Kobarid

Answer 6

I'm wondering if the PR_SET_PDEATHSIG flag is getting cleared, even though you set it after you fork (and before exec), so it seems from the docs like it shouldn't get cleared.

In order to test that theory, you could try the following: use the same code to run a subprocess that's written in C and basically just calls prctl(PR_GET_PDEATHSIG, &result) and prints the result.

Another thing you might try: adding explicit zeros for arg3, arg4, and arg5 when you call prctl. I.e.:

>>> implant_bomb = lambda: libc.prctl(PR_SET_PDEATHSIG, TERM, 0, 0, 0)

Slick

Answer 7

There is some security restriction to take into account because if we call setuid after execv he child cannot receive signal. The complete list of this restrictions is here

good luck !
/Mohamed

Colin

Interesting, so the signal gets delivered only if parent process has sufficient privileges to send signals to child processes. Typically any child process running with higher privilege than its parent will receive no signal. I haven't noticed such a thing. How can I detect if it somehow increases its privilege? – Kobarid

Answer 8

Other answers mention prctl's PR_SET_DEATHSIG but leave out the fact that this can be set from the command line using the setpriv command:

setpriv --pdeathsig HUP [command] &

Lancaster