RuntimeError: This event loop is already running in python

Question

I think I'm getting this error because my code calls asyncio.get_event_loop().run_until_complete(foo()) twice. Once from foo() and second time from function called by foo(). My question is then: why should this be a problem? Why should I even care that this loop is running?

---

There was an edit made to this question which, I think, obscured it (some people prefer to follow rules without understanding them, thus an "illegal" word was removed from the title). Unfortunately, this creates confusion.

I'm not surprised by the fact that the error is raised. I can trace it back to the asyncio source and see that the authors of this library wanted to do it this way, there's no mystery there. The puzzling part is in the reason the authors of the library decided it's illegal to ask from event loop to run some function to completion when the loop is already running.

We can reduce the problem to just two such calls, and through case analysis we will see that these are the three possibilities:

1. Neither of both functions ever terminates.
2. One of the functions eventually terminates.
3. Both functions eventually terminate.

Now, is there any sane behavior which would address all three cases? To me, it is obvious that there is, or, perhaps are multiple sane behaviors possible here. For example:

1. Nothing special, the execution of both functions is interleaved, and they keep running forever, just as expected.
2. The loop doesn't return control to the code following the first instance of run_until_complete() until second function completes (thus no code after run_until_complete() will be executed.
3. After the last function terminates, the loop returns control to the first code object which invoked run_until_complete ignoring all other invocation sites.

Now, I can understand that this behavior may not be something that everyone would want. But, since this library decided to give programmers control over starting / stopping the event loop, it should also meet the consequences of such decisions. Making it an error to start the same loop multiple times precludes library code from ever doing this, which reduces the quality and usefulness of libraries utilizing asyncio (which is indeed the case with, for example, aiohttp).

Filterable

Answer 1

I got the issue resolved by using the nest_async

pip install nest-asyncio

and adding below lines in my file.

import nest_asyncio
nest_asyncio.apply()
# EDIT: the __import__ line is not needed, leaving it just 
# so that the comments make sense
# __import__('IPython').embed() 

Galligaskins

This answer would be better if it explained what the given code snippet actually does, especially if you contrast it with the code in the question – Interrelated

Brilliant! @DeanGurvitz This above code monkey patches the asyncio event loop and allows it to be re-entrant (you may calll run_until_complete while run_until_complete is already on the stack). – Beautifully

Agree with @DeanGurvitz – Livelong

Don't underestimate this trick's stability, since its used to run Jupyter Notebooks at scale in Netflix' papermill package (papermill.execute_notebook(nest_asyncio=True) – Rellia

@Rellia Big adoption does not mean big stability and just furthers cargo culting. – Annora

@MaximilianBurszley Good thing nest_asyncio and Python are both open source! – Flambeau

I was running into this issue as well and this resolved it. Thanks! – Intermediate

I think we don't need __import__('IPython').embed(). An example usage related to this would be helpful – Meurer

Installing this package is absolutely unnecessary, the accepted answer should be the one below that simply suggests doing asyncio.create_task – Calzada

Great not every function will take as long as the function that takes the longest to complete – Jovia

this was magic. – Hopson

@Meurer is right, just the first two lines of the answer will do the job. __import__('IPython').embed() is completely unnecessary. – Martingale

Answer 2

Event loop running - is an entry point of your async program. It manages running of all coroutines, tasks, callbacks. Running loop while it's running makes no sense: in some sort it's like trying to run job executor from same already running job executor.

Since you have this question, I guess you may misunderstand a way how asyncio works. Please, read this article - it's not big and gives a good introduction.

Upd:

There's absolutely no problem in adding multiple things to be ran by event loop while this loop is already running. You can do it just by awaiting for it:

await coro()  # add coro() to be run by event loop blocking flow here until coro() is finished

or creating a task:

# python 3.7+
asyncio.create_task(coro())  # add coro() to be run by event loop without blocking flow here

# This works in all Python versions but is less readable
asyncio.ensure_future(coro())

As you can see you don't need call event loop's methods to make something being ran by it.

Event loop's method such as run_forever or run_until_complete — are just a ways to start event loop in general.

run_until_complete(foo()) means: "add foo() to be ran by event loop and run event loop itself until foo() isn't done".

Monostich

...but I'm not running the event loop, I'm running foo() why do I even have to run the loop explicitly? Who on Earth would want that? It's like moving the hands of your wall clock with your hands... to make time move forward... – Filterable

Re' article - thank you. I'll read it now. Compared to any language which put concurrency into its design from the get go, asyncio is so bad, on so many levels... that it seems beyond redemption. Everything is bad, the design, the implementation, the documentation... Python was overall a decent language, but this addition... I don't even know how to describe it. – Filterable

Well, the article is very superficial, with couple hello-world level examples which don't hold water when applied to something more realistic... :/ – Filterable

@Filterable I really think it contains all needed stuff to see how to use asyncio. If you want something more deep, to understand how asyncio itself works, I can advice you to watch this video: youtube.com/watch?v=MCs5OvhV9S4 It's more complex, but it shows how event loop (abstract one) organized and how it manages things. – Monostich

I don't even know where to begin to describe all the problems this article glosses over. I think the person who wrote it simply never had a need for any sort of concurrency and just wrote it as a "theoretical" exercise. Anyways, nothing in this article or in your answer explains why is it a problem to try to run something on the same loop while something else is running. You think that it doesn't make sense, because you restrict yourself to a very unrealistic case when you only have a very simple program, with a single entry point and no libraries. No useful program is like that. – Filterable

Well, there actually is a problem, which is by design of asyncio... The problem is that the programmer should never start / stop running the loop. This is just inherently bad idea. Now, since the loop needs to be started, there's a problem: who starts it? Can library code start it? It needs to be started from non-async code, perhaps multiple times. What if neither place knows anything about the other place and just wants to start the loop? To answer this, you need to explain why asyncio throws when two different functions try to start it. Just stating the fact that it does it isn't helpful – Filterable

asyncio.ensure_future(coro()) was the solution for me 👍 – Whitehead

in some contexts, like a decator in simple unittest.Testcase methods, it works fine. But the other answers above say that it will only work correctly at top level. If you are in some other context, including as I discovered running in a 3.6+ ready ipython which already has built in async handling, it blows out with the questions error. I may not like it but I see why it would be so. – Procumbent

I don't see any theoretical reason why synchronous code shouldn't be able to submit work to the event queue and wait for a response. It really seems like an implementation oversight in Python. Not allowing re-entrance of the event loop has the toxic side-effect that any code that needs to call and async function must itself be async (unless its running in another thread). – Beautifully

See Ovidiu Ionut's answer. Event loops can be reentrant. That is it not by default was a religious design descision that needlessly forks async from sync code. bugs.python.org/issue22239 – Beautifully

@Beautifully that any code that needs to call and async function must itself be async - it's by design. Please, read this answer. nest_asyncio is needed rare cases where environmet creates event loop for you like in Jupyter. It doesn't eliminate need to mark functions that use async code as async itself in general. – Monostich

I think nest_asyncio (or similar) is needed whenever you want to introduce an asyncio component to a legacy project. Async-await's implementation (with the deault evenet loop) prohibits incremental refactoring (partitioning sync from async universes). One call to an async function means that 30 functions above it in the stack need to be touched with async-await, regardless of whether they care about I/O (e.g. timing decorators, memoization decorators, unit test code .... all must be made async aware). – Beautifully

await coro() only works from within a coroutine, which is an important caveat – Kovach

await coro() is so neat and solved my problem without installing any external packages. Thanks! – Submediant

Answer 3

Just add this bunch of code in the beginning

!pip install nest_asyncio
import nest_asyncio
nest_asyncio.apply()

Variegated

I consider this not to be good advice. There is no explanation about what this code does, why it even solves the problem and what side effects it might have. – Helicopter

Citing nest_asyncio: By design asyncio does not allow its event loop to be nested. This presents a practical problem: When in an environment where the event loop is already running it’s impossible to run tasks and wait for the result. Trying to do so will give the error “RuntimeError: This event loop is already running”. ... This module patches asyncio to allow nested use of asyncio.run and loop.run_until_complete. Source: pypi.org/project/nest-asyncio – Jagatai

Answer 4

I'm writing this down not to patronize, but to explain how we can handle the situation where simply queueing async functions and awaiting their results synchronously while the event loop is running, doesn't work.

run_until_complete is not for running any number of arbitrary async functions synchronously, it is for running the main entry point of your entire async program. This constraint is not immediately apparent from the docs.

Since libraries like aiohttp will queue it's own entry point to run as a server and block the loop's synchronous operations using run_until_complete or run_forever, the event loop will already be running and you won't be able to run independent synchronous operations on that event loop and wait for it's result within that thread.

That being said, if you have to queue an async operation into a running event loop from within a sync context and get it's result like a regular function, that may not be possible. One way is to pass in a synchronous callback to be called once the async operation finishes. That will of course slow down your event loop. Another way is to put the result in a queue and wait for it in a synchronous loop, emulating the event loop.

Another way of handling the situation is to execute your code within startup and cleanup callbacks of the async http library you're using. Here's a sample of how you may accomplish this.

Trigeminal

Answer 5

Using nest_asyncio didn't work for me, because then aiohttp started complaining with

• RuntimeError: Timeout context manager should be used inside a task

Instead I decided to replace all calls to asyncio.run with calls to this asyncio_run:

def asyncio_run(future, as_task=True):
    """
    A better implementation of `asyncio.run`.

    :param future: A future or task or call of an async method.
    :param as_task: Forces the future to be scheduled as task (needed for e.g. aiohttp).
    """

    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:  # no event loop running:
        loop = asyncio.new_event_loop()
        return loop.run_until_complete(_to_task(future, as_task, loop))
    else:
        nest_asyncio.apply(loop)
        return asyncio.run(_to_task(future, as_task, loop))


def _to_task(future, as_task, loop):
    if not as_task or isinstance(future, Task):
        return future
    return loop.create_task(future)

A secondary goal was to be able to think of asyncio.run as promise.resolve from the JS world, or Task.Wait from the .NET world.

---

Edit: In our product we removed this code in favor of this other answer in a related question.

Sike

Am I missing something here, or is this junk syntax? what is else doing? – Woollen

@JamieMarshall no it's not junk, try-except-else works so that if the try part succeeds, then the else is executed after it. So it's like an else for the except. It allows having only the maybe-raises part in try, and the rest of the normal flow in the else. docs.python.org/3/tutorial/errors.html#handling-exceptions – Seadog

Answer 6

I had the same problem and I fixed thsi issue using nest_async or asyncio

Solution #1

Simply install the package :

pip install nest-asyncio

Then add these lines:

import nest_asyncio
nest_asyncio.apply()

Solution #2

If it's not working with nest_asyncio, then try asyncio :

import asyncio
asyncio.set_event_loop(asyncio.new_event_loop())

Spherics

Answer 7

Sometimes the event loop is there and is running. In that case we don't need to run it again, we just need to create a task using create_task

try this

try:
    loop = asyncio.get_event_loop()
except RuntimeError:
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
        
task = loop.create_task(user_insert_events(target))
if not loop.is_running():
    loop.run_until_complete(task)

Detwiler

if the loop is running though, how would you wait until the task is finished and get the result out synchronously? – Pylorectomy

Answer 8

Here is a general async-to-sync context manager I came up with based on this answer:

await def async_func():
   ...

def sync_func():
    with sync_await() as await_:
       x = await_(async_func())

It can actually be useful even in async context, e.g. for converting an async iterable to a regular (sync) iterable:

await def async_func(coroutines):
   with sync_await() as await_:
        # TypeError: 'async_generator' object is not iterable
        # x = any(await c for c in coroutines)
        x = any(map(await_, coroutines))

Heck

Answer 9

This might not solve the same problem but a similar one. If the error is not produced by any of your custom files but by packages you install using pip, proceed to read below because that was how I fixed mine.

I realized this error was due to package versions and compatibility. So you might want to look at the stack trace and determine which libraries are involved and try upgrading or downgrading them, keeping in mind their compatibility. If you are lucky enough pip install might inform you of any incompatibilities.

Once again, in my case, I had to upgrade to

PyMySQL==1.1.0
aiomysql==0.2.0

Foresight

Answer 10

I found that changing this: filtered_cap = pyshark.FileCapture( <FILE_NAME>.pcap)

To this: with pyshark.FileCapture( <FILE_NAME>.pcap) as filtered_cap:

Resolves the Cannot run the event loop while another loop is running Error

Kare