I want to do parallel http request tasks in asyncio, but I find that python-requests would block the event loop of asyncio. I've found aiohttp but it couldn't provide the service of http request using a http proxy.
So I want to know if there's a way to do asynchronous http requests with the help of asyncio.
To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:
import asyncio
import requests
@asyncio.coroutine
def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = yield from future1
response2 = yield from future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
This will get both responses in parallel.
With python 3.5 you can use the new await/async syntax:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = await future1
response2 = await future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
See PEP0492 for more.
SyntaxError: 'yield' outside function –
Langur requests.get will be executing in another thread. I believe one of the big pros of asyncio is the idea of keeping things single-threaded: not having to deal with shared memory, locking, etc. I think my confusion lies in the fact that your example uses both asyncio and concurrent.futures module. –
Fransis hooks= keyword when calling Requests like that, as hook functions will likely break the self-contained requirement, depending on what you do in there. –
Khartoum loop = asyncio.get_event_loop() appears twice? –
Rubeola run_in_executor doesn't allow to pass kwargs for target callback func (like loop.run_in_executor(None, requests.get, 'http://www.google.com', auth=blah) # fails, this can be achieved with lambda or functools.partial as a proxy: loop.run_in_executor(None, lambda: requests.get('http://www.google.com', auth=blah)). see python.org/dev/peps/pep-3156/#callback-style –
Hydrocele main() need to be in event loop when the main() body already running event loop? –
Bartender main() isn't running an event loop. It gets the event loop in order to run requests.get() asynchronously. If a function wants to await an async function, it needs to be running in the event loop. –
Unknow do_get() that calls requests.get() and await that function? As far as I understand, the program will go do something else while it waits for do_get() to finish, and the only thing do_get() does is make the requests call. Won't that be async then? –
Caporal requests.get() isn't async, so when do_get() would call requests.get(), the function would block. The event loop will only do something else when a function yields control (using await) which requests.get() doesn't. And since it's blocking, the only way to run something else concurrently is in another thread/process, which is what run_in_executor() does. –
Unknow aiohttp can be used with HTTP proxy already:
import asyncio
import aiohttp
@asyncio.coroutine
def do_request():
proxy_url = 'http://localhost:8118' # your proxy address
response = yield from aiohttp.request(
'GET', 'http://google.com',
proxy=proxy_url,
)
return response
loop = asyncio.get_event_loop()
loop.run_until_complete(do_request())
The answers above are still using the old Python 3.4 style coroutines. Here is what you would write if you got Python 3.5+.
aiohttp supports http proxy now
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def main():
urls = [
'http://python.org',
'https://google.com',
'http://yifei.me'
]
tasks = []
async with aiohttp.ClientSession() as session:
for url in urls:
tasks.append(fetch(session, url))
htmls = await asyncio.gather(*tasks)
for html in htmls:
print(html[:100])
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
There is also the httpx library, which is a drop-in replacement for requests with async/await support. However, httpx is somewhat slower than aiohttp.
Another option is curl_cffi, which has the ability to impersonate browsers' ja3 and http2 fingerprints.
Requests does not currently support asyncio and there are no plans to provide such support. It's likely that you could implement a custom "Transport Adapter" (as discussed here) that knows how to use asyncio.
If I find myself with some time it's something I might actually look into, but I can't promise anything.
There is a good case of async/await loops and threading in an article by Pimin Konstantin Kefaloukos Easy parallel HTTP requests with Python and asyncio:
To minimize the total completion time, we could increase the size of the thread pool to match the number of requests we have to make. Luckily, this is easy to do as we will see next. The code listing below is an example of how to make twenty asynchronous HTTP requests with a thread pool of twenty worker threads:
# Example 3: asynchronous requests with larger thread pool
import asyncio
import concurrent.futures
import requests
async def main():
with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
loop = asyncio.get_event_loop()
futures = [
loop.run_in_executor(
executor,
requests.get,
'http://example.org/'
)
for i in range(20)
]
for response in await asyncio.gather(*futures):
pass
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
for i in range(10000) because one requests might fail or timeout, right? –
Saucepan requests does not support asyncio so you need to create threads to get concurrency. –
Simpkins Considering that aiohttp is fully featured web framework, I’d suggest to use something more light weighted like httpx (https://www.python-httpx.org/) which supports async requests. It has almost identical api to requests:
>>> async with httpx.AsyncClient() as client:
... r = await client.get('https://www.example.com/')
...
>>> r
<Response [200 OK]>
python-requests does not natively support asyncio yet. Going with a library that natively supports asyncio, like httpx would be the most beneficial approach.
However, if your use cases heavily relies on using python-requests you can wrap the sync calls with asyncio.to_thread and asyncio.gather and follow the asyncio programming patterns.
import asyncio
import requests
async def main():
res = await asyncio.gather(asyncio.to_thread(requests.get("YOUR_URL"),)
if __name__ == "__main__":
asyncio.run(main())
For concurrency/parallelization of the network requests:
import asyncio
import requests
urls = ["URL_1", "URL_2"]
async def make_request(url: string):
response = await asyncio.gather(asyncio.to_thread(requests.get(url),)
return response
async def main():
responses = await asyncio.gather((make_request(url) for url in urls))
for response in responses:
print(response)
if __name__ == "__main__":
asyncio.run(main())
DISCLAMER: Following code creates different threads for each function.
This might be useful for some of the cases as it is simpler to use. But know that it is not async but gives illusion of async using multiple threads, even though decorator suggests that.
To make any function non blocking, simply copy the decorator and decorate any function with a callback function as parameter. The callback function will receive the data returned from the function.
import asyncio
import requests
def run_async(callback):
def inner(func):
def wrapper(*args, **kwargs):
def __exec():
out = func(*args, **kwargs)
callback(out)
return out
return asyncio.get_event_loop().run_in_executor(None, __exec)
return wrapper
return inner
def _callback(*args):
print(args)
# Must provide a callback function, callback func will be executed after the func completes execution !!
@run_async(_callback)
def get(url):
return requests.get(url)
get("https://google.com")
print("Non blocking code ran !!")
© 2022 - 2025 — McMap. All rights reserved.
subprocessto parallel your your code. – Amalekiterequests(likegoogle-auth) and can't be trivially rewritten to useaiohttp. – Consequent