python asyncio exceptions raised from loop.create_task()

Question

I want my code to use python logging to log exceptions. In my usual code using await, exceptions are raised normally, so:

try: await code_that_can_raise() except Exception as e: logger.exception("Exception happended")

Works fine.

However, when using loop.create_task(coro())

I'm not sure how can I catch the exception here.
Wrapping the create_task() call obviously won't work. What is the best solution to log every exception in the code?

Unconditioned

Answer 1

What is the best solution to log every exception in the code?

If you control the invocation of create_task, but don't control the code in the coro(), then you can write a logging wrapper:

async def log_exceptions(awaitable):
    try:
        return await awaitable
    except Exception:
        logger.exception("Unhandled exception")

then you can call loop.create_task(log_exceptions(coro())).

If you can't or don't want to wrap every create_task, you can call loop.set_exception_handler, setting the exception to your own function that will log the exception as you see fit.

September

It's better to use except Exception: instead of except: to avoid suppressing KeyboardInterrupt and similar non-errors. – Sachsse

Any general solution that is based on the event loop? rather than wrapping every create_task call. Since as I understand, the exception is eventually raised up to the event loop. – Unconditioned

@Unconditioned You can use set_exception_handler for that purpose; see the editer answer. – September

how do you await in a function that is not async? – Stereotyped

@Stereotyped Good catch, I've now added the missing async. – September

Can't we instead await the returned Task from "create_task" and then check its error state? Because here we seem limited to logging the error but we can't raise it upper – Agon

@EricBurel Immediately awaiting the task returned by create_task would defeat the purpose of calling create_task in the first place, which is to run the task in the background (of sorts). I.e. await create_task(foo()) would be no better than await foo(). if you just write create_task(foo()), it means you want to run foo() independently of what you're doing. There are several ways how you could propagate an exception raised by foo(), and the advice in this answer is tailored to the needs of the question. – September

Make sense, I've opened a separate question: #76614667 – Agon

Answer 2

Just so that it has been mentioned: asyncio.Task objects have the methods result and exception.
result:

[...] if the coroutine raised an exception, that exception is re-raised [...]

exception:

[...] If the wrapped coroutine raised an exception that exception is returned [...]

Given a simple setup (in Python 3.7 syntax):

import asyncio
tasks =[]

async def bad_test():
    raise ValueError

async def good_test():
    return

async def main():
    tasks.append(asyncio.create_task(bad_test()))
    tasks.append(asyncio.create_task(good_test()))

asyncio.run(main())

Using result, one could do:

for t in tasks:
    try:
        f = t.result()
    except ValueError as e:
        logger.exception("we're all doomed")

Or, using exception:

for t in tasks:
    if isinstance(t.exception(), Exception):
        logger.exception("apocalypse now")

However, both methods require the Task to be done, otherwise:

If the Task has been cancelled, this method raises a CancelledError exception.

(result): If the Task’s result isn’t yet available, this method raises a InvalidStateError exception.

(exception): If the Task isn’t done yet, this method raises an InvalidStateError exception.

So, unlike the proposal in the other answer, the logging will not happen when the exceptions raise in the tasks, but rather when the tasks are evaluated after they completed.

Amr

What about a task that is not expected to complete? I have a task that is supposed to consume a queue forever, it finishes when we cancel it – Agon

I've tried a solution with "add_done_callback", however when raising an exception from there it doesn't "bubble up", I can log it but it is not caught upper – Agon

Answer 3

Expanding on @user4815162342's solution, I created a wrapper around log_exceptions to avoid having to nest every coroutine inside 2 functions:

import asyncio
from typing import Awaitable

def create_task_log_exception(awaitable: Awaitable) -> asyncio.Task:
    async def _log_exception(awaitable):
        try:
            return await awaitable
        except Exception as e:
            logger.exception(e)
    return asyncio.create_task(_log_exception(awaitable))

Usage:

create_task_log_exception(coroutine())

Felike

Answer 4

The proper way is to use create_task but you need await it if you want to catch the exception at some point:

import asyncio

async def sleepError(x):
  await asyncio.sleep(x)
  print(1)
  throw_error = 1 / 0

async def sleepOk(x):
  await asyncio.sleep(x)
  print(2)

async def main():
  x = asyncio.create_task(sleepError(1))
  await sleepOk(2)
  
  """
  await x
  # print(3) bellow works without "await x", and print(1) from sleepError as well
  # You can try/except the "await x" line
  # if you "await x" without try/except, print(3) is not executed but error happens.
  # if you don't "await x" you get warning: Task exception was never retrieved
  """
  print(3)


asyncio.run(main())

Belda

Answer 5

If you want to react to the exception of a task as soon as it occurs you can use add_done_callback() ( https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.add_done_callback )

asyncio.Task objects is an asyncio.Future like object and has the add_done_callback() method.

In the callback function you just have to get the result() of the future to provoke an exception. with try except you can add custom handling / logging whatever.

import asyncio

async def sleepError(x):
  await asyncio.sleep(x)
  print(1)
  throw_error = 1 / 0

def done_callback(futr):
    try:
        rslt = futr.result()
    except Exception as exc:
        # do_something_with(exc) if you want to (like logging)
        # or just raise
        raise

async def sleepOk(x):
  await asyncio.sleep(x)
  print(2)

async def main():
  x = asyncio.create_task(sleepError(1))
  # without next line exception will only occur at end of main
  x.add_done_callback(done_callback)
  await sleepOk(2)
  print(3)

asyncio.run(main())

If you just want to see the exception on the console, then following callback is sufficient:

def done_callback(futr):
    rslt = futr.result()

Simard