In Micropython, how can I guarantee asynchronous tasks being shut down?

Note: despite I'm beginning my question with a microdot example, this is just because I've first observed the later described behavior with an attempt to re-start a microdot server after aborting a running async script with CTRL-C. Chances are that a solution to my problem doesn't have anything to to with microdot, but I'm not sure so I keep the example for now.

Update: this is not an ESP32 issue neither. Running the snippets using plain Micropython gives the same results. (see quick installation guide at end of this post)

Update: to me it feels like a but so I made a report

I'm struggling with the following short async snippet running on MicroPython v.1.23 on an ESP32 device:

import asyncio
from microdot import Microdot

async def amain():
    app = Microdot()

    @app.route('/')
    async def index(request):
        return 'Hello, world!!'

    await app.start_server(host="0.0.0.0", port=80)

asyncio.run(amain())

The code does what you'd expect it to do (i.e. nothing, because the device is not reachable due to WiFi not being set up, but that's not important for now). Unfortunately it works only the first time after each reboot.

Trying to run it the second time results in

File "main.py", line 26, in amain                                                                         
File "microdot.py", line 1234, in start_server                                                              
File "asyncio/stream.py", line 1, in start_server                                                           
OSError: [Errno 112] EADDRINUSE

due to the fact the asyncio server doesn't terminate, which seems to be a known issue.

However, that doesn't seem to be a problem with the server itself, since you can still shut it down manually, e.g. this way:

    task = asyncio.create_task(app.start_server(host="0.0.0.0", port=80))
    await asyncio.sleep(5)
    task.cancel()

now the server shuts down after 5 seconds, unconditionally.

Easy thing, I thought, just catch KeyboardInterrupt or CancelledError and shut down the server using cancel() and you're done with only a little workaround.

Here the asyncio-only (microdot-agnostic) part begins

Some steps further - and without the web server anymore - I came up with a script that tries to handle a shutdown of the main application in order to cancel() my problematic task:

import asyncio

async def async_foo():
    try:
        while True:
            print("bar")
            await asyncio.sleep(1)
    except asyncio.CancelledError:
        print("async_foo:CancelledError")
    except KeyboardInterrupt:
        print("async_foo:KeyboardInterrupt")
    finally:
        print("async_foo:finally")

async def amain():
    task = asyncio.create_task(async_foo())
    try:
        await task
    except asyncio.CancelledError:
        print("amain:CancelledError")
    except KeyboardInterrupt:
        print("amain:KeyboardInterrupt")
    finally:
        print("amain:finally")

def run():
    try:
        asyncio.run(amain())
    except asyncio.CancelledError:
        print("run:CancelledError")
    except KeyboardInterrupt:
        print("run:KeyboardInterrupt")
    finally:
        print("run:finally")

run()

Here I'm catching all sorts of exceptions in order to find the one I need to handle, only to see those exceptions not being propagated to the async tasks as I'd expect.

Running this snippet on CPython (3.12 for me) I get

$ python3 -c "import main; main.run()"
bar
bar
bar
^Casync_foo:CancelledError
async_foo:finally
amain:finally
run:finally

while on MicroPython only run() will catch exceptions:

import main; main.run()
bar
bar
run:KeyboardInterrupt
run:finally

So what's going on here? It looks like MicroPython asyncio immediately shuts down all tasks without raising exceptions, but unfortunately asyncio.run_server, which gets invoked by microdot keeps running for some reason.

Is this a bug? How can I let a coroutine handle CancelledError in general (resp. how would I shutdown microdot.start_server()/asyncio.start_server() specificly)?

Install Micropython (on Ubuntu/Debian):

# maybe install some requirements sudo apt-get install build-essential libffi-dev git pkg-config
git clone https://github.com/micropython/micropython.git
cd micropython/ports/unix
make submodules
make -j8
build-standard/micropython [<PATH-TO-FILE>]

Handled Exceptions

The below code will never return a KeyboardInterrupt as the error is handled already. Calling for an interruption will automatically try-except and close() the asyncio loop. But we may want to try a few things out first...

import asyncio from microdot import Microdot async def amain(): app = Microdot() @app.route('/') async def index(request): return 'Hello, world!!' await app.start_server(host="0.0.0.0", port=80) # ← Does not return KeyboardInterrupt. asyncio.run(amain())

The question is, does the loop ever terminate the first time?

1: If it does, but somehow leaves the asyncio event loop with a running thread, try to close it after it leaves the await.

await app.start_server(host="0.0.0.0", port=80) app.shutdown() # ← calls `close()` on the *event loop*.

EDIT: shutdown() calls close() which tries to close the event loop. So if a thread stays alive despite the event-loop being terminated, shutdown() does nothing for us.

2: If it never leaves the loop and gets stuck. Try to implement a remote shutdown to see if it works better than interrupts (not a definitive answer, but informative).

@app.route('/shutdown') def shutdown(request): request.app.shutdown() return 'The server is shutting down...'

And paste http://localhost/shutdown in your web browser or cmd→curl.

Issue

Handled Exceptions

Alternative explanation

Recommended topics

Hot tags