Why am I getting NotImplementedError with async and await on Windows?

Question

I have this code:

import os
import time
import asyncio


async def run_command(*args):
    """
    Example from:
        http://asyncio.readthedocs.io/en/latest/subprocess.html
    """
    # Create subprocess
    process = await asyncio.create_subprocess_exec(
        *args,
        # stdout must a pipe to be accessible as process.stdout
        stdout=asyncio.subprocess.PIPE)

    # Wait for the subprocess to finish
    stdout, stderr = await process.communicate()

    # Result
    result = stdout.decode().strip()

    # Return stdout
    return result


def run_asyncio_commands(tasks):
    """Run tasks using asyncio and return results"""
    loop = asyncio.get_event_loop()
    commands = asyncio.gather(*tasks)  # Unpack list using *
    results = loop.run_until_complete(commands)
    loop.close()
    return results


if __name__ == '__main__':

    start = time.time()

    cmds = [
        ['du', '-sh', '/Users/fredrik/Desktop'],
        ['du', '-sh', '/Users/fredrik'],
        ['du', '-sh', '/Users/fredrik/Pictures']
    ]

    tasks = []
    for cmd in cmds:
        tasks.append(run_command(*cmd))
    results = run_asyncio_commands(tasks)
    print(results)

    end = time.time()
    print('Script ran in', str(end - start), 'seconds')

When I run the that code in Python 3.6.1 on my mac, I get this:

['780K\t/Users/fredrik/Desktop', '46G\t/Users/fredrik', '52M\t/Users/fredrik/Pictures']
Script ran in 6.405519008636475 seconds

But when I run the same script on Windows (but with the du commands substituted to something which works on Windows), also with Python 3.6.1, I get this:

Traceback (most recent call last):
  File "C:\Users\iruser\Desktop\asynciotest.py", line 66, in <module>
    results = run_asyncio_commands(tasks)
  File "C:\Users\iruser\Desktop\asynciotest.py", line 41, in run_asyncio_commands
    results = loop.run_until_complete(commands)
  File "C:\Users\fredrik\condaenvs\dev_py36\lib\asyncio\base_events.py", line 466, in run_until_complete
    return future.result()
  File "C:\Users\iruser\Desktop\asynciotest.py", line 16, in run_command
    stdout=asyncio.subprocess.PIPE)
  File "C:\Users\fredrik\condaenvs\dev_py36\lib\asyncio\subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "C:\Users\fredrik\condaenvs\dev_py36\lib\asyncio\base_events.py", line 1190, in subprocess_exec
    bufsize, **kwargs)
  File "C:\Users\fredrik\condaenvs\dev_py36\lib\asyncio\coroutines.py", line 210, in coro
    res = func(*args, **kw)
  File "C:\Users\fredrik\condaenvs\dev_py36\lib\asyncio\base_events.py", line 340, in _make_subprocess_transp
ort
    raise NotImplementedError
NotImplementedError

This is what I substitute the Unix commands with on Windows:

cmds = [['C:/Windows/system32/HOSTNAME.EXE']]

Python and Windows version info:

Python 3.6.1 | packaged by conda-forge | (default, May 23 2017, 14:21:39) [MSC v.1900 64 bit (AMD64)] on win32
Windows 10 Pro, version 1703, OS build 15063.413

Grappling

Answer 1

Different event loops are implemented differently. Some of them have restrictions (sometimes OS-related). By default, Windows uses SelectorEventLoop and as you can see in doc:

SelectorEventLoop has the following limitations:

• SelectSelector is used to wait on socket events: it supports sockets and is limited to 512 sockets.
• loop.add_reader() and loop.add_writer() only accept socket handles (e.g. pipe file descriptors are not supported).
• Pipes are not supported, so the loop.connect_read_pipe() and loop.connect_write_pipe() methods are not implemented.
• Subprocesses are not supported, i.e. loop.subprocess_exec() and loop.subprocess_shell() methods are not implemented.

To run your code in Windows you can use alternative event loop available by default - ProactorEventLoop.

Replace line:

loop = asyncio.get_event_loop()

with this:

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

Your code will work.

Merth

By adding: asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy()) – Brashy

How can I use the ProactorEventLoop when running tests with pytest and pytest-asyncio? – Navicert

To the question in my comment above @pytest.fixture \n def event_loop(request): \n loop = asyncio.ProactorEventLoop() \n yield loop \n loop.close() – Navicert

@Navicert you should probably create a custom event_loop. Take a look here - github.com/pytest-dev/pytest-asyncio#event_loop – Merth

this did not work for me - i added the two lines from the solution and the policy from Jordan and it still says NotImplementedError for loop.add_reader – Fortunato

Answer 2

3.7.0 Python documentation handles this here: https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess

Set the event loop policy if you are using Windows - then your code will work.

In your startup, change the unix-specific section:

cmds = [
    ['du', '-sh', '/Users/fredrik/Desktop'],
    ['du', '-sh', '/Users/fredrik'],
    ['du', '-sh', '/Users/fredrik/Pictures']
]

to handle Windows & Unix:

if 'win32' in sys.platform:
    # Windows specific event-loop policy & cmd
    asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
    cmds = [['C:/Windows/system32/HOSTNAME.EXE']]
else:
    # Unix default event-loop policy & cmds
    cmds = [
        ['du', '-sh', '/Users/fredrik/Desktop'],
        ['du', '-sh', '/Users/fredrik'],
        ['du', '-sh', '/Users/fredrik/Pictures']
    ]

Brashy

Note that WindowsSelectorEventLoopPolicy and WindowsProactorEventLoopPolicy were added in Python 3.7 first, so setting default policy won't work with Python 3.6 or earlier. – Standardize

Answer 3

This has a GitHub issue page [https://github.com/saghul/aiodns/issues/86][1]

The latest addition is a suggestion to add the following:

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Hope this helps someone.

Aylmer

Answer 4

I tested in python and ipython, it seems the loop can be change in python, but not in ipython, and I also found the default loop is `` in jupyter and can not change neither.

Tests in python

C:\Users\Liu.D.H>python
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsProactorEventLoopPolicy object at 0x0000018054ACC790>
>>> asyncio.get_event_loop()
<stdin>:1: DeprecationWarning: There is no current event loop
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.get_event_loop()
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsSelectorEventLoopPolicy object at 0x0000018054A8E0E0>
>>> asyncio.get_event_loop()
<_WindowsSelectorEventLoop running=False closed=False debug=False>
>>> exit()

C:\Users\Liu.D.H>

Tests in ipython

C:\Users\Liu.D.H>ipython
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import asyncio

In [2]: asyncio.get_event_loop_policy()
Out[2]: <asyncio.windows_events.WindowsProactorEventLoopPolicy at 0x1df598b7e20>

In [3]: asyncio.get_event_loop()
<ipython-input-3-6908e23590ee>:1: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop()
Out[3]: <ProactorEventLoop running=False closed=False debug=False>

In [4]: asyncio.get_event_loop()
<ipython-input-4-6908e23590ee>:1: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop()
Out[4]: <ProactorEventLoop running=False closed=False debug=False>

In [5]: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

In [6]: asyncio.get_event_loop_policy()
Out[6]: <asyncio.windows_events.WindowsSelectorEventLoopPolicy at 0x1df59a6c820>

In [7]: asyncio.get_event_loop()
<ipython-input-7-6908e23590ee>:1: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop()
Out[7]: <ProactorEventLoop running=False closed=False debug=False>

In [8]: asyncio.get_event_loop()
<ipython-input-8-6908e23590ee>:1: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop()
Out[8]: <ProactorEventLoop running=False closed=False debug=False>

In [9]: asyncio.set_event_loop(asyncio.SelectorEventLoop())

In [10]: asyncio.get_event_loop()
<ipython-input-10-6908e23590ee>:1: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop()
Out[10]: <ProactorEventLoop running=False closed=False debug=False>

In [11]:

Melvamelvena

Answer 5

Jupyter on Windows uses "SelectorEventLoop", to make asyncio.subprocess work you must use "ProactorEventLoop" but there's no way to do it.

There's GitHub's issue (2) that's been open since 2019, and it probably won't get fixed anytime soon (1) (2).

However I managed to run a docker process asynchronously from jupyter using subprocess.Popen:

import asyncio
import subprocess


async def run_docker(args: list[str]):
    def run():
        process = subprocess.Popen(
            ["docker"]
            + args,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
        )

        stdout, stderr = process.communicate()

        if stderr:
            raise Exception(stderr)

        return stdout

    return await asyncio.to_thread(run)

Personally, I find it quite disappointing that as of 2023, the Python ecosystem still lacks a strong and universally adopted method for running processes asynchronously. Even JavaScript managed to address this issue quite a while back.

Burl

Answer 6

remove --reload from your uvicorn run

The --reload option is useful for development as it automatically reloads the server when code changes are detected, but it can interfere with how the event loop and subprocesses are managed, especially when using libraries like Playwright.

Aspirator