Why child process (daemon=True) not exiting when main process exit in python?

Question

Here is the official explanation of daemon flag in python multiprocessing:

When a process exits, it attempts to terminate all of its daemonic child processes.

By my understanding, the parent process will kill its children whose daemon flag is set to be True when it exits.

Below is the code I used to prove my guess. But the result is different.

import multiprocessing


def child():
    while True:
        pass


for x in xrange(1, 4):
    proc = multiprocessing.Process(target=child, args=())
    proc.daemon=True
    proc.start()


while True:
    pass

The above starts 4 child processes and one main process. I killed the main process but the 4 children did not exit.

So why are they not terminated by main since the daemon is set to be true?

Routine

Which OS are you on? When I try on Windows, I get a bunch of process related errors. – Skiagraph

I ran this on my Linux system and have added an answer, check it out. – Skiagraph

Answer 1

Notes:

• The use of xrange the implies Python 2

• xrange(1, 4) will yield 3 values not 4 (so, there will only be 3 children)

This is not quite how things work. The doc ([Python 2.Docs]: multiprocessing - daemon) should probably be more specific.

The thing is that multiprocessing registers a cleanup function to kill all its deamonic children when exiting. That is done via [Python 2.Docs]: atexit - Exit handlers (emphasis is mine):

Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

You don't handle the TERM signal (sent by default by the kill command), therefore the cleanup function is not called by the main process (leaving its children running).

I modified your code to better illustrate the behavior.

code00.py:

#!/usr/bin/env python2

import multiprocessing
import os
import sys
import time


print_text_pattern = "Output from process {:s} - pid: {:d}, ppid: {:d}"


def child(name):
    while True:
        print(print_text_pattern.format(name, os.getpid(), os.getppid()))
        time.sleep(1)


def main(*argv):
    procs = list()
    for x in xrange(1, 3):
        proc_name = "Child{:d}".format(x)
        proc = multiprocessing.Process(target=child, args=(proc_name,))
        proc.daemon = True #x % 2 == 0
        print("Process {:s} daemon: {:}".format(proc_name, proc.daemon))
        procs.append(proc)

    for proc in procs:
        proc.start()

    counter = 0
    while counter < 3:
        print(print_text_pattern.format("Main", os.getpid(), os.getppid()))
        time.sleep(1)
        counter += 1


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.\n")
    sys.exit(rc)

Notes:

• Changed the way how children processes are spawned a bit: all of them are created 1^st, and only then started

• Added some print calls from each process, to track their activity in the StdOut - also added some time.sleep calls (1 second), to avoid producing too much output

• Most important - the main process no longer runs forever. At some point it exits gracefully (after 3 cycles - due to counter variable), and there's when the behavior that I mentioned earlier kicks in.
  This could also have been possible by intercepting the TERM signal (and others that can be explicitly be sent by the kill command) and performing the cleanup then - in that way the children would be killed as well when killing the main process - but that's more complicated

• I simplified things a bit so that only 2 children are spawned

• Moved everything in a main function (for structure) enclosed in a if __name__ == "__main__": conditional, so the processes are not spawned if you import the module

• Give different values proc.daemon for each child then monitor the output and ps -ef | grep "code00.py" output

• Added an argument (name) to child func, but that's only for display purposes

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow]> python2 ./code00.py
Python 2.7.12 (default, Oct  8 2019, 14:14:10) [GCC 5.4.0 20160609] 064bit on linux2

Process Child1 daemon: True
Process Child2 daemon: True
Output from process Main - pid: 1433, ppid: 1209
Output from process Child1 - pid: 1434, ppid: 1433
Output from process Child2 - pid: 1435, ppid: 1433
Output from process Main - pid: 1433, ppid: 1209
Output from process Child2 - pid: 1435, ppid: 1433
Output from process Child1 - pid: 1434, ppid: 1433
Output from process Main - pid: 1433, ppid: 1209
Output from process Child1 - pid: 1434, ppid: 1433
Output from process Child2 - pid: 1435, ppid: 1433
Output from process Child1 - pid: 1434, ppid: 1433
Output from process Child2 - pid: 1435, ppid: 1433

Done.

Irizarry

Very well explained! Take my upvote :) So in short, you mean to say, that OP is not handling the TERM signal, when they let the main process run indefinitely until they abruptly kill it? And that is why the cleanup function for child processes are not called? – Skiagraph

@nj2237: Thank you. Yes, if the OP would handle TERM, and inside the handler would call the cleanup function that multiprocessing registers, the code from the question would behave as expected. – Irizarry

Okay, thanks for the clarification. So calling sys.exit() in the main process also handles TERM, which is why it worked for me. – Skiagraph

@Skiagraph I don't claim that. sys.exit gracefully exits the process, and the atexit funcs are executed. I am not sure whether it sends the signal or not (didn't dig that deep into the code). – Irizarry

okay, I will edit my answer back. Also I checked this - pymotw.com/2/atexit/index.html and looks like calling sys.exit will also register the atexit callbacks. – Skiagraph

Answer 2

Yes, your understanding is correct and your code to test this also works.

I just added some sleep statements to debug the output (without sleep, it is difficult to infer from the huge output of prints):

import multiprocessing
import time
import sys

print("main")

def child():
    while True:
        print("child")
        time.sleep(3)

for x in xrange(1, 4):
    proc = multiprocessing.Process(target=child, args=())
    proc.daemon=True
    proc.start()

time.sleep(7)
print("exit")
sys.exit() # this exits the main process

Now, when I ran this script, and when it was running, I did a ps aux and could see four processes running from this script. After 7 seconds, when I did ps aux again, I could no more see those processes running - which means:

When the main process exited, it terminated all of its daemonic child processes.

After that, I also set the proc.daemon to False, and ran the script once again. This time, even after 7 seconds, when I did a ps aux, I could still see the child processes running (since they are non-daemonic now, they do not exit even after main process terminates).

So this works as expected - let me know if you still have an issue here.

Edit 1: Thanks to @CristiFati for pointing out the original cleanup issue. This code works because calling sys.exit() also registers the atexit callbacks as elaborated here.

Skiagraph