What is the right way to await cancelling an asyncio task?

Question

The docs for cancel make it sound like you should usually propagate CancelledError exceptions:

Therefore, unlike Future.cancel(), Task.cancel() does not guarantee that the Task will be cancelled, although suppressing cancellation completely is not common and is actively discouraged. Should the coroutine nevertheless decide to suppress the cancellation, it needs to call Task.uncancel() in addition to catching the exception.

However, neither of the methods for detecting cancellation are awaitable: cancelling which tells you if cancelling is in progress, and cancelled tells you if cancellation is done. So the obvious way to wait for cancellation is this:

foo_task.cancel()
try:
    await foo_task
except asyncio.CancelledError:
    pass

There are lots of examples of this online even on SO. But the docs warn you asyncio machinery will "misbehave" if you do this:

The asyncio components that enable structured concurrency, like asyncio.TaskGroup and asyncio.timeout(), are implemented using cancellation internally and might misbehave if a coroutine swallows asyncio.CancelledError

Now you might be wondering why you would wait to block until a task is fully cancelled. The problem is the asyncio event loop only creates weak references to tasks, so if as your class is shutting down (e.g. due to a cleanup method or __aexit__) and you don't await every task you spawn, you might tear down the only strong reference while the task is still running, and then python will yell at you:

ERROR base_events.py:1771: Task was destroyed but it is pending!

So it seems to avoid the error I am specifically being forced into doing the thing I'm not supposed to do :P The only alternative seems to be weird unpythonic hackery like stuffing every task I make in a global set and awaiting them all at the end of the run.

Crucible

Just out of curioscity - why would you want to cancel an async task? Async frameworks are not usually designed to start and stop tasks like a task scheduling system might do. – Gabrielegabriell

@FreelanceConsultant that's a weird question, asyncio is definitely designed to support task cancellation, I linked to several sections in the docs discussing it. – Crucible

@FreelanceConsultant but a really simple example is supporting operations that timeout, or because you're doing something on behalf of a user that's taking awhile and they click a cancel button – Crucible

In the first case, should the asyncronous task just resolve into the error state? Regarding user interaction, I'm not sure that fits well with all asynchronous libraries. Maybe you're right that this is an intended use case - I am not sure. – Gabrielegabriell

@FreelanceConsultant ⁤Another example might be when used within lifespan context for a FastAPI app. Lets say we use RabbitMQ. ⁤⁤It has durable queues and it's crucial to ensure that each message is processed. Closing these tasks is essential to avoid any loss of messages during shutdown. ⁤ – Trimetallic

Answer 1

You have a cleanup problem. The with statement is generally used to solve cleanup.

Use asyncio.TaskGroup:

async with asyncio.TaskGroup() as tg:
    tg.create_task(some_coro(...)).cancel()

Materi

Answer 2

You can use gather to wrap the task in an awaitable that will conclude without raising a CancelledError after the cancelled task finishes:

task = asyncio.create_task(some_coroutine())
group = asyncio.gather(task)
task.cancel()
await group

This is an alternative for Python versions prior to 3.11, where asyncio.TaskGroup was introduced.

Haiti

This doesn't appear to actually work (Python 3.10): gist.github.com/sersorrel/1e3e0f40ee5c46388b6dd6e45f803260 – Estuarine

...but it does seem to work with return_exceptions=True. – Estuarine

The documentation specifically says that gather won't be cancelled by a CancelledError, so I really don't know why this is happening. – Haiti