concurrent.futures.ThreadPoolExecutor swallowing exceptions (Python 3.6)

Question

I'm trying to use ThreadPoolExecutor in Python 3.6 on Windows 7 and it seems that the exceptions are silently ignored or stop program execution. Example code:

#!/usr/bin/env python3

from time import sleep

from concurrent.futures import ThreadPoolExecutor

EXECUTOR = ThreadPoolExecutor(2)


def run_jobs():
    EXECUTOR.submit(some_long_task1)
    EXECUTOR.submit(some_long_task2, 'hello', 123)
    return 'Two jobs was launched in background!'


def some_long_task1():
    print("Task #1 started!")
    for i in range(10000000):
        j = i + 1
    1/0
    print("Task #1 is done!")


def some_long_task2(arg1, arg2):
    print("Task #2 started with args: %s %s!" % (arg1, arg2))
    for i in range(10000000):
        j = i + 1
    print("Task #2 is done!")


if __name__ == '__main__':
    run_jobs()
    while True:
        sleep(1)

The output:

Task #1 started!
Task #2 started with args: hello 123!
Task #2 is done!

It's hanging there until I kill it with Ctrl+C.

However, when I remove 1/0 from some_long_task1, Task #1 completes without problem:

Task #1 started!
Task #2 started with args: hello 123!
Task #1 is done!
Task #2 is done!

I need to capture the exceptions raised in functions running in ThreadPoolExecutor somehow.

Python 3.6 (Minconda), Windows 7 x64.

Westnorthwest

Answer 1

ThreadPoolExecutor.submit returns a future object that represents the result of the computation, once it's available. In order to not ignore the exceptions raised by the job, you need to actually access this result. First, you can change run_job to return the created futures:

def run_jobs():
    fut1 = EXECUTOR.submit(some_long_task1)
    fut2 = EXECUTOR.submit(some_long_task2, 'hello', 123)
    return fut1, fut2

Then, have the top-level code wait for the futures to complete, and access their results:

import concurrent.futures

if __name__ == '__main__':
    futures = run_jobs()
    concurrent.futures.wait(futures)
    for fut in futures:
        print(fut.result())

Calling result() on a future whose execution raised an exception will propagate the exception to the caller. In this case the ZeroDivisionError will get raised at top-level.

Locular

Answer 2

You can handle the exceptions with a try statement. This is how your some_long_task1 method could look like:

def some_long_task1():
    print("Task #1 started!")
    try:
        for i in range(10000000):
            j = i + 1
        1/0
    except Exception as exc:
        print('some_long_task1 generated an exception: {}'.format(exc))
    print("Task #1 is done!")

Output when the method is used within your script:

Task #1 started!
Task #2 started with args: hello 123!
some_long_task1 generated an exception: integer division or modulo by zero
Task #1 is done!
Task #2 is done!
(the last while loop running...)

Fleurette

This does not really answer the question since the question is targeted at catching exception of a subthread and from the nature of the Q description it does not seem like understanding try: catch: is the problem here, since the OP is asking about catching exceptions specifically already. – Lavellelaven

Answer 3

As previous answers indicated, there are two ways to catch the exceptions for the ThreadPoolExecutor - check the exception in the future or log them. It all depends on what one wants to deal with the potential exceptions. In general, my rule of thumb includes:

1. If I only want to record the error, including the trace stack. log.exception is a better choice. It needs to add extra logic to record the same amount of information via future.exception().
2. If the code needs to handle different exceptions differently in the main thread. Checking the status of of future is the way to go. Also, you might find the function as_completed is also useful in this circumstance.

Odell