how limit request per second with httpx [Python 3.6]

Question

My project consists of consuming an api that is built on top of the aws lambda service. Technically, the leader who built the api tells me that there is no fixed request limit since the service is elastic, but it is important to take into account the number of requests per second that the api can support.

To control the limit of requests per second (concurrently), the python script that I am developing uses asyncio and httpx to consume the api concurrently, and taking advantage of the max_connections parameter of httpx.Limits I am trying to find the optimal value so that the api does not freeze.

My problem is that I don't know if I am misinterpreting the use of the max_connections parameter, since when testing with a value of 1000, my understanding tells me that per second I am making 1000 requests concurrently to the api, but even so, the api after a certain time freezes.

I would like to be able to control the limit of requests per second without the need to use third-party libraries.

How could I do it?

Here is my MWE

async def consume(client, endpoint: str = '/create', reg):
  data = {"param1": reg[1]}

  response = await client.post(url=endpoint, data=json.dumps(data))

  return response.json()

async def run(self, regs):
  # Empty list to consolidate all responses
  results = []

  # httpx limits configuration
  limits = httpx.Limits(max_keepalive_connections=None, max_connections=1000)
  timeout = httpx.Timeout(connect=60.0, read=30.0, write=30.0, pool=60.0)

  # httpx client context
  async with httpx.AsyncClient(base_url='https://apiexample', headers={'Content-Type': 'application/json'},
                              limits=limits, timeout=timeout) as client:
    
    # regs is a list of more than 1000000 tuples
    tasks = [asyncio.ensure_future(consume(client=client, reg=reg))
            for reg in regs]
   
    result = await asyncio.gather(*tasks)
    results += result
       
  return results

Thanks in advance.

Torin

Be careful not to conflate 'connections' with 'requests'. If the API is throttled at, say, '1000 requests per second', having 1000 connections at once can easily go over that limit if connection reuse takes place (i.e., each connection being used to send more than one request). To really limit the number of "in-flight" requests I suggest to use asyncio.semaphore right before the await client.post line. – Shulock

Answer 1

Your leader is wrong - there is a request limit for AWS lambda (it's 1000 concurrent executions by default).

AWS API is highly unlikely to "freeze" (there are many layers of protection), so I would look for a problem on your side. Start debugging, by lowering the concurent connections setting (e.g. 100), and explore other settings if this doesn't fix the issue..

More info: https://www.bluematador.com/blog/why-aws-lambda-throttles-functions

Deron