Using asyncio for Non-async Functions in Python?

Question

Suppose there is a library that makes various database queries:

import time

def queryFoo():
    time.sleep(4)
    return "foo"

def queryBar():
    time.sleep(4)
    return "bar"

I want to execute those 2 queries concurrently without having to add async to the method signature or adding a decorator. These functions should not depend on asyncio at all.

What is the best way to utilize those non-async functions within asyncio?

I am looking for something of the form:

#I need an 'asyncWrapper'

results = asyncio.gather(asyncWrapper(queryFoo()), asyncWrapper(queryBar()))

Thank you in advance for your consideration and response.

Weatherboarding

Answer 1

If some function is blocking and not async by nature, only proper way to run it inside asyncio event loop is to run it inside thread using run_in_executor:

# Our example blocking functions
import time


def queryFoo():
    time.sleep(3)
    return 'foo'


def queryBar():
    time.sleep(3)
    return 'bar'


# Run them using asyncio
import asyncio
from concurrent.futures import ThreadPoolExecutor


_executor = ThreadPoolExecutor(10)


async def in_thread(func):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_executor, func)


async def main():
    results = await asyncio.gather(
        in_thread(queryFoo), 
        in_thread(queryBar),
    )

    print(results)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

It does job.

If you however want to avoid using threads only way to do it - is to rewrite queryFoo/queryBar to be async by nature.

Bluefish

Coming from here, the example you showed above will still need async function for run_in_executor(), while I want to make sure the async function converted to sync such that I don't have nested async calls that have nested event loops. One of the solutions I can think of is to spawn new event loop for the inner async calls and block until its result got returned, and continue to next outer loops. Will that work? – Koniology

@EmersonXu 1) if you want to cast some async function to sync, you just have to do loop.run_until_complete(asyncfunc()) inside sync funtions. But once you did it, you've done with async world at all: from now on all your outer code is sync. 2) If you want to cast some sync function to async, use await loop.run_in_executor(_executor, syncfunc) inside other async functions. 3) If you want to do something else, I'm afraid it's hard to me to understand it without reproducible code example. – Bluefish

If queryFoo has input (keyword) arguments, I can add a line in in_thread to do func = partial(func, *args, **kwargs), or is there a better way? (and partial comes from from functools import partial). – Fatshan

@BertCoerver partial - is a proper way to handle such situation. – Bluefish

Answer 2

I presume you are after concurrency and hopefully do not insist on using asyncio module itself in which case this little example could be helpful:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

def queryFoo():
    time.sleep(2)
    return "FOO"

def queryBar():
    time.sleep(4)
    return "BAR"

with ThreadPoolExecutor(max_workers=2) as executor:
    foo = executor.submit(queryFoo)
    bar = executor.submit(queryBar)
    results = [foo.result(), bar.result()]

print(results)

It runs both queryFoo() and queryBar() in parallel and collects their results in a list in order in which they've been mentioned in an assignment to results.

Occultation

Looks like he updated the question to say he does need it to be an "asyncio wrapper". – Cacogenics