how to get response_time and response_size while using aiohttp

Question

Is it possible to get response time and response size for each request made using aiohttp?

The documentation seems not to have those properties anywhere.

Thanks

Countrybred

Answer 1

One possibility might be:

• measure point in time before request
• measure point in time after request
• the difference is the response time
• with 'response.text()' you get the response and can determine the length with 'len()'

A small self-contained example could look like this:

import time
import asyncio
from aiohttp import ClientSession


async def fetch(session, url):
    start = time.time()
    async with session.get(url) as response:
        result = await response.text()
        end = time.time()
        print(url, ": ", end - start, "response length:", len(result))
        return result


async def crawl(urls: set):
    async with ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(
                fetch(session, url)
            )
        await asyncio.gather(*tasks)


if __name__ == "__main__":
    urlSet = {"https://www.software7.biz/tst/number.php",
              "https://www.software7.biz/tst/number1.php",
              "https://www.software7.biz"}
    asyncio.run(crawl(urlSet))

Test

The two endpoints number.php and number1.php have a delay on server side of 3 respective 1 second and are returning a two digit number each.

The output in the debug console looks like this then:

https://www.software7.biz :  0.16438698768615723 response length: 4431
https://www.software7.biz/tst/number1.php :  1.249755859375 response length: 2
https://www.software7.biz/tst/number.php :  3.214473009109497 response length: 2

Punt

By this approach response time will be affected by asyncio event loop because session.get() task can wait some amount of time in the event loop queue before execution – Oxblood

Answer 2

len(response.text()) will return size of decompressed response. If you want the size of the raw compressed response you need to set auto_decompress=False during creation of aiohttp.ClientSession. After that you can get it with len(await response.read()). But it'll make response.text() unavailable since it needs uncompressed response. To make it available again you'll have to decompress it manually:

import time
import zlib
import brotli

async with aiohttp.ClientSession(auto_decompress=False) as session:
    start = time.monotonic()
    response = await session.get(url='www.test.com')
    response_time = time.monotonic() - start
    response_size = len(await response.read())

    encoding = response.headers['Content-Encoding']
    if encoding == 'gzip':
        response._body = zlib.decompress(response._body, 16 + zlib.MAX_WBITS)
    elif encoding == 'deflate':
        response._body = zlib.decompress(response._body, -zlib.MAX_WBITS)
    elif encoding == 'br':
        response._body == brotli.decompress(response._body)

    response_text = await response.text()

About time.time() from pymotw.com:

Because time.time() looks at the system clock, and the system clock can be changed by the user or system services for synchronizing clocks across multiple computers, calling time.time() repeatedly may produce values that go forwards and backwards. This can result in unexpected behavior when trying to measure durations or otherwise use those times for computation. Avoid those situations by using time.monotonic(), which always returns values that go forward.

aiohttp docs suggest to use loop.time() (which is also monotonic):

async def on_request_start(session, trace_config_ctx, params):
    trace_config_ctx.start = asyncio.get_event_loop().time()

async def on_request_end(session, trace_config_ctx, params):
    elapsed = asyncio.get_event_loop().time() - trace_config_ctx.start
    print("Request took {}".format(elapsed))

trace_config = aiohttp.TraceConfig()
trace_config.on_request_start.append(on_request_start)
trace_config.on_request_end.append(on_request_end)
async with aiohttp.ClientSession(trace_configs=[trace_config]) as client:
    client.get('http://example.com/some/redirect/')

Fletafletch

Nice answer, but an incorrect one w.r.t. timing response. The aiohttp loads just the headers with the session.get() call and returns control. One perhaps needs to time the calls to content loading methods like response.read or .text or .json. Adding them together gives the actual 'response time', but it may not be so trivial, unless both these calls are together, and they're timed together. – Tsarina

If the connection count is limited (e.g. connector = aiohttp.TCPConnector(limit=10)), the Request took time computed by the above code includes the time waiting for other connections to finish (or time out). You can use on_connection_create_start instead of on_request_start to begin the timer only when the connection is initiated. – Petronel

on_connection_create_start and on_connection_create_end called only if opening new connection to server. Most subsequent requests will be waiting in queue firing on_connection_queued_start and on_connection_queued_end and then on_connection_reuseconn signals instead – Ivonne

Answer 3

You can get the size of the response content from the headers:

response.headers['content-length']

Economically