Python Multiprocessing atexit Error "Error in atexit._run_exitfuncs"

Question

I am trying to run a simple multiple processes application in Python. The main thread spawns 1 to N processes and waits until they all done processing. The processes each run an infinite loop, so they can potentially run forever without some user interruption, so I put in some code to handle a KeyboardInterrupt:

#!/usr/bin/env python
import sys
import time
from multiprocessing import Process

def main():
    # Set up inputs..

    # Spawn processes
    Proc( 1).start()
    Proc( 2).start()

class Proc ( Process ):
    def __init__ ( self, procNum):
        self.id = procNum
        Process.__init__(self)

    def run ( self ):
        doneWork = False

        while True:

            try:
                # Do work...
                time.sleep(1)
                sys.stdout.write('.')

                if doneWork:
                    print "PROC#" + str(self.id) + " Done."
                    break

            except KeyboardInterrupt:
                print "User aborted."
                sys.exit()

# Main Entry
if __name__=="__main__":
    main()

The problem is that when using CTRL-C to exit, I get an additional error even though the processes seem to exit immediately:

......User aborted.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python26\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python26\lib\multiprocessing\util.py", line 281, in _exit_function
    p.join()
  File "C:\Python26\lib\multiprocessing\process.py", line 119, in join
    res = self._popen.wait(timeout)
  File "C:\Python26\lib\multiprocessing\forking.py", line 259, in wait
    res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Python26\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python26\lib\multiprocessing\util.py", line 281, in _exit_function
    p.join()
  File "C:\Python26\lib\multiprocessing\process.py", line 119, in join
    res = self._popen.wait(timeout)
  File "C:\Python26\lib\multiprocessing\forking.py", line 259, in wait
    res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt

I am running Python 2.6 on Windows. If there is a better way to handle multiprocessing in Python, please let me know.

Voltaism

Answer 1

Rather then just forcing sys.exit(), you want to send a signal to your threads to tell them to stop. Look into using signal handlers and threads in Python.

You could potentially do this by changing your while True: loop to be while keep_processing: where keep_processing is some sort of global variable that gets set on the KeyboardInterrupt exception. I don't think this is a good practice though.

Novotny

If possible, can you add a runnable example of this solution. – Purposive

Also looking for an elaboration on the usage of signal handlers – Fidele

Answer 2

This is a very old question, but it seems like the accepted answer does not actually fix the problem.

The main issue is that you need to handle the keyboard interrupt in the parent process as well. Additionally to that, in the while loop, you just need to exit the loop, there's no need to call sys.exit()

I've tried to as closely match the example in the original question. The doneWork code does nothing in the example so have removed that for clarity.

import sys
import time
from multiprocessing import Process


def main():
    # Set up inputs..

    # Spawn processes
    try:
        processes = [Proc(1), Proc(2)]
        [p.start() for p in processes]
        [p.join() for p in processes]
    except KeyboardInterrupt:
        pass


class Proc(Process):
    def __init__(self, procNum):
        self.id = procNum
        Process.__init__(self)

    def run(self):
        while True:
            try:
                # Do work...
                time.sleep(1)
                sys.stdout.write('.')

            except KeyboardInterrupt:
                print("User aborted.")
                break


# Main Entry
if __name__ == "__main__":
    main()

Apograph