Send Simultaneous Requests python (all at once)

Question

I'm trying to create a script that send's over 1000 requests to one page at the same time. But requests library with threading (1000) threads. Seems to be doing to first 50 or so requests all within 1 second, whereas the other 9950 are taking considerably longer. I measured it like this.

def print_to_cmd(strinng):
    queueLock.acquire()
    print strinng
    queueLock.release()

    start = time.time()
    resp = requests.get('http://test.net/', headers=header)
    end = time.time()

    print_to_cmd(str(end-start))

I'm thinking requests library is limiting how fast they are getting sent.

Doe's anybody know a way in python to send requests all at the same time? I have a VPS with 200mb upload so that is not the issue its something to do with python or requests library limiting it. They all need to hit the website within 1 second of each other.

Thanks for reading and I hope somebody can help.

Misreport

Are you trying to overload a site? – Staphylo

Nobody is going to help you DDOS a website. – Toddler

If I wanted to DDOS a website I would use multiple servers with shells. – Misreport

You might look into BoundedSemaphore (or for Python 2). It may be more flexible-- in terms of concurrency throughput-- than a simple lock. – Mcconnell

Answer 1

I have generally found that the best solution is to use an asynchronous library like tornado. The easiest solution that I found however is to use ThreadPoolExecutor.

---

import requests
from concurrent.futures import ThreadPoolExecutor

def get_url(url):
    return requests.get(url)
with ThreadPoolExecutor(max_workers=50) as pool:
    print(list(pool.map(get_url,list_of_urls)))

Huttan

Sure, you may want to play around with the max_workers parameter to get faster run times – Huttan

yes I have noticed 1k requests takes considerably longer. but still, it is better than what I had before – Misreport

There is typo brah, I did correct it: print(list(pool.map(get_url(list_of_urls))) – Subdivide

@Enderphan Nope, check out how map works: docs.python.org/3/library/functions.html#map ThreadPoolExecutor().map() is the same idea. – Reprobate

with post it gives to me error requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) – Dhow

@Dhow you might want to add a try/catch there and maybe some error handling/retries. That means that your http request was unsuccessful. – Huttan

Answer 2

I know this is an old question, but you can now do this using asyncio and aiohttp.

import asyncio
import aiohttp
from aiohttp import ClientSession

async def fetch_html(url: str, session: ClientSession, **kwargs) -> str:
    resp = await session.request(method="GET", url=url, **kwargs)
    resp.raise_for_status()
    return await resp.text()

async def make_requests(url: str, **kwargs) -> None:
    async with ClientSession() as session:
        tasks = []
        for i in range(1,1000):
            tasks.append(
                fetch_html(url=url, session=session, **kwargs)
            )
        results = await asyncio.gather(*tasks)
        # do something with results

if __name__ == "__main__":
    asyncio.run(make_requests(url='http://test.net/'))

You can read more about it and see an example here.

Jackpot

asyncio.run is a Python 3.7 addition. For previous versions, refer to this discussion – Fib

Answer 3

Assumed that you know what you are doing, I first suggest you to implement a backoff policy with a jitter to prevent "predictable thundering hoardes" to your server. That said, you should consider to do some threading

import threading
class FuncThread(threading.Thread):
    def __init__(self, target, *args):
        self._target = target
        self._args = args
        threading.Thread.__init__(self)

    def run(self):
        self._target(*self._args)

so that you would do something like

t = FuncThread(doApiCall, url)
t.start()

where your method doApiCall is defined like this

def doApiCall(self, url):

Klina

Hi thanks for the fast response. I used threading in the previous attempt but sent the requests using requests library for python. does this request without having to use urllib or requests? – Misreport