KeyboardInterrupt with Python multiprocessing.Pool

Question

I want to write a service that launches multiple workers that work infinitely and then quit when main process is Ctrl+C'd. However, I do not understand how to handle Ctrl+C correctly.

I have a following testing code:

import os
import multiprocessing as mp
    

def g():
    print(os.getpid())
    while True:
        pass
        
        
def main():
    with mp.Pool(1) as pool:
        try:
            s = pool.starmap(g, [[]] * 1)
        except KeyboardInterrupt:
            print('Done')


if __name__ == "__main__":
    print(os.getpid())
    main()

When I try to Ctrl+C it, I expect process(es) running g to just receive SIGTERM and silently terminate, however, I receive something like that instead:

Process ForkPoolWorker-1:
Done
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "test.py", line 8, in g
    pass
KeyboardInterrupt

This obviously means that parent and children processes both raise KeyboardInterrupt from Ctrl+C, further suggested by tests with kill -2. Why does this happen and how to deal with it to achieve what I want?

Weisburgh

Does this answer your question? Keyboard Interrupts with python's multiprocessing Pool – Incrassate

@Charchit No, the question you linked is concerned with a bug in old python version which has been fixed since then, mine is not related to that. – Weisburgh

The accepted answer talks about the bug, the other answers are more recent and this one (on the same thread) basically says the same thing as the accepted answer here – Incrassate

Answer 1

The signal that triggers KeyboardInterrupt is delivered to the whole pool. The child worker processes treat it the same as the parent, raising KeyboardInterrupt.

The easiest solution here is:

1. Disable the SIGINT handling in each worker on creation
2. Ensure the parent terminates the workers when it catches KeyboardInterrupt

You can do this easily by passing an initializer function that the Pool runs in each worker before the worker begins doing work:

import signal
import multiprocessing as mp

# Use initializer to ignore SIGINT in child processes
with mp.Pool(1, initializer=signal.signal, initargs=(signal.SIGINT, signal.SIG_IGN)) as pool:
    try:
        s = pool.starmap(g, [[]] * 1)
    except KeyboardInterrupt:
        print('Done')

The initializer replaces the default SIGINT handler with one that ignores SIGINT in the children, leaving it up to the parent process to kill them. The with statement in the parent handles this automatically (exiting the with block implicitly calls pool.terminate()), so all you're responsible for is catching the KeyboardInterrupt in the parent and converting from ugly traceback to simple message.

Ranjiv

Using initializer for ignoring SIGINT is elegant, I will likely use it. But why is it delivered to the whole pool? When I tested it with kill -2 <parent pid>, it simply behaved as I expected, only sending it to parent and printing Done. Is this bash behavior? – Weisburgh

@steam_engine: Per this answer, "The SIGINT signal is generated by the terminal line discipline, and broadcast to all processes in the terminal's foreground process group. Your shell has already created a new process group for the command (or command pipeline) that you ran, and told the terminal that that process group is its (the terminal's) foreground process group." Basically, yes, hitting Ctrl-C, at least on POSIX, sends SIGINT to the entire foreground process group, not just the parent process. – Ranjiv

Presumably os.setpgrp() could work too (untested, don't quote me), if the child process initializer used it to detach from the parent process group, but it's UNIX-only, so if you want consistent behavior across OSes, it's not viable. – Ranjiv