Script stuck on exit when using atexit to terminate threads

Question

I'm playing around with threads on python 3.7.4, and I want to use atexit to register a cleanup function that will (cleanly) terminate the threads.

For example:

# example.py
import threading
import queue
import atexit
import sys

Terminate = object()

class Worker(threading.Thread):
    def __init__(self):
        super().__init__()
        self.queue = queue.Queue()

    def send_message(self, m):
        self.queue.put_nowait(m)

    def run(self):
        while True:
            m = self.queue.get()
            if m is Terminate:
                break
            else:
                print("Received message: ", m)


def shutdown_threads(threads):
    for t in threads:
        print(f"Terminating thread {t}")
        t.send_message(Terminate)
    for t in threads:
        print(f"Joining on thread {t}")
        t.join()
    else:
        print("All threads terminated")

if __name__ == "__main__":
    threads = [
        Worker()
        for _ in range(5)
    ]
    atexit.register(shutdown_threads, threads)

    for t in threads:
        t.start()

    for t in threads:
        t.send_message("Hello")
        #t.send_message(Terminate)

    sys.exit(0)

However, it seems interacting with the threads and queues in the atexit callback creates a deadlock with some internal shutdown routine:

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
^CException ignored in: <module 'threading' from '/usr/lib64/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 1308, in _shutdown
    lock.acquire()
KeyboardInterrupt
Terminating thread <Worker(Thread-1, started 140612492904192)>
Terminating thread <Worker(Thread-2, started 140612484511488)>
Terminating thread <Worker(Thread-3, started 140612476118784)>
Terminating thread <Worker(Thread-4, started 140612263212800)>
Terminating thread <Worker(Thread-5, started 140612254820096)>
Joining on thread <Worker(Thread-1, stopped 140612492904192)>
Joining on thread <Worker(Thread-2, stopped 140612484511488)>
Joining on thread <Worker(Thread-3, stopped 140612476118784)>
Joining on thread <Worker(Thread-4, stopped 140612263212800)>
Joining on thread <Worker(Thread-5, stopped 140612254820096)>
All threads terminated

(the KeyboardInterrupt is me using ctrl-c since the process seems to be hanging indefinitely).

However, if I send the Terminate message before exit(uncomment the line after t.send_message("Hello")), the program doesn't hang and terminates gracefully:

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Terminating thread <Worker(Thread-1, stopped 140516051592960)>
Terminating thread <Worker(Thread-2, stopped 140516043200256)>
Terminating thread <Worker(Thread-3, stopped 140515961992960)>
Terminating thread <Worker(Thread-4, stopped 140515953600256)>
Terminating thread <Worker(Thread-5, stopped 140515945207552)>
Joining on thread <Worker(Thread-1, stopped 140516051592960)>
Joining on thread <Worker(Thread-2, stopped 140516043200256)>
Joining on thread <Worker(Thread-3, stopped 140515961992960)>
Joining on thread <Worker(Thread-4, stopped 140515953600256)>
Joining on thread <Worker(Thread-5, stopped 140515945207552)>
All threads terminated

This begs the question, when does this threading._shutdown routine gets executed, relative to atexit handlers? Does it make sense to interact with threads in atexit handlers?

Jackal

Why don't you want to do #t.send_message(Terminate)? – Derain

Apparently the interpreter doesn't call the atexit handlers until all the non-daemon threads have exited, which suspiciously sounds like a bug that was fixed in Python 2.6.5 (see - #3713860 and bugs.python.org/issue1722344). A workaround might be to wrap the main code in try / finally and manually call shutdown_threads(threads) yourself. – Multiply

Also see How to terminate a thread when main program ends? – Multiply

Answer 1

You can use one daemon thread to ask your non-daemon threads to clean up gracefully. For an example where this is necessary, if you are using a third-party library that starts a non-daemon thread, you'd either have to change that library or do something like:

import threading

def monitor_thread():
    main_thread = threading.main_thread()
    main_thread.join()
    send_signal_to_non_daemon_thread_to_gracefully_shutdown()


monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()

start_non_daemon_thread()

To put this in the context of the original poster's code (note we don't need the atexit function, since that won't get called until all the non-daemon threads are stopped):

if __name__ == "__main__":
    threads = [
        Worker()
        for _ in range(5)
    ]
    
    for t in threads:
        t.start()

    for t in threads:
        t.send_message("Hello")
        #t.send_message(Terminate)

    def monitor_thread():
        main_thread = threading.main_thread()
        main_thread.join()
        shutdown_threads(threads)

    monitor = threading.Thread(target=monitor_thread)
    monitor.daemon = True
    monitor.start()

Tooley

This worked as expected for me but I noticed that it also works if monitor_thread is not a daemon thread. My explanation is that it already waits on main_thread.join() and therefore will wake up when main_thread exits. The docs say "Daemon threads are abruptly stopped at shutdown.", which makes me think that here we actually may want monitor not be a daemon thread. – Cleat

Here's a HUGE upvote and thank you. I've been struggling with this exact problem all day, where I want to start a quick thread (loading some data) upon django 4.2 startup. For some cryptic reason, I faced the exact same problem with the local development mode with autoreloader (using sys.exit(3) upon reloading), and skipping using atexit and signal handlers was the single best solution that worked. – Orinasal

Answer 2

atexit.register(func) registers func as a function to be executed at termination.

After execute the last line of code (it is sys.exit(0) in above example) in main thread, threading._shutdown was invoked (by interpreter) to wait for all non-daemon threads (Workers created in above example) exit

The entire Python program exits when no alive non-daemon threads are left.

So after typing CTRL+C, the main thread was terminated by SIGINT signal, and then atexit registered functions are called by interpreter.

By the way, if you pass daemon=True to Thread.__init__, the program would run straightforward without any human interactive.

Fortunate

Yes, but I want the threads to be killed gracefully, giving them a chance to execute cleanup code. That's what my atexit handler is supposed to be for. – Jackal

As said above, you could achieve it via making the Worker a daemon thread (super().__init__(daemon=True). – Fortunate

It's my understanding that daemon threads are not given a chance to handle their termination gracefully, e.g. cleanup any resources they might be holding on. They are simply killed brutally when the main thread exits, and are not considered at all during the runtime shutdown process. See this for potential issues with using them, for example: joeshaw.org/python-daemon-threads-considered-harmful – Jackal

atexit is an exception to allow us do some cleanup actions before Python interpreter do real finalization. > The interpreter is still entirely intact at this point (github.com/python/cpython/blob/master/Python/…) – Fortunate

As a prove, you could add a logging after Worker got a Terminate from queue. You would see that the Workers are still alive during the execution of atexit registered functions. – Fortunate

Ok, I see, that make sense. Either I use daemon threads and atexit, or I have to do shutdown "manually" before exit. Thanks! – Jackal