aiohttp how to save a persistent ClientSession in a class?

Question

I'm writing a class that will do http requests using aiohttp. According to the docs I should not to create a ClientSession per request, so I want to reuse the same session.

code:

class TestApi:
   def __init__(self):
      self.session = aiohttp.ClientSession()

   # async defs methods from here 

When doing

TestApi()

I get the error: Unclosed client session.

What is the solution to persist the ClientSession object?

Asyut

Answer 1

The expression TestApi() on a line by itself creates a TestApi object and immediately throws it away. aiohttp complaints that the session was never closed (either by leaving an async with block or with an explicit call to close()), but even without the warning it doesn't make sense not to assign the API object to a variable where it will be actually used.

To reuse the session, your code needs to have access to the session, or to an object that holds it:

async def fetch(url):
    async with aiohttp.request('GET', url) as resp:
        resp.raise_for_status()
        return await resp.read()

async def main():
    url1_data, url2_data = asyncio.gather(
        fetch('http://url1'), fetch('http://url2'))
    url3_data, url4_data = asyncio.gather(
        fetch('http://url3'), fetch('http://url4'))

One option is to add a session parameter to fetch (and other functions) and consistently call it with a session created in main(). A better option is to create an API class and convert the global functions like fetch to methods:

class Http:
    async def __aenter__(self):
        self._session = aiohttp.ClientSession()
        return self

    async def __aexit__(self, *err):
        await self._session.close()
        self._session = None

    async def fetch(self, url):
        async with self._session.get(url) as resp:
            resp.raise_for_status()
            return await resp.read()

main() can still exist as a function, but it can consistently use the object that holds the session:

async def main():
    async with Http() as http:
        url1_data, url2_data = await asyncio.gather(
            http.fetch('http://url1'), http.fetch('http://url2'))
        url3_data, url4_data = await asyncio.gather(
            http.fetch('http://url3'), http.fetch('http://url4'))

In the above code, the async with statement is used to ensure that the session is closed whenever the scope is left.

Dell

Is it possible to detect when the Http class is destroyed (go out of scope) so I can manually close the session? – Asyut

@Asyut Not easily, because session.close() needs to be awaited (which the initial version of my answer neglected to do, fixed now). The recommended way is to use async with, so you don't forget to close it regardless of how you exit the function. – Dell

I've now edited the answer to demonstrate the usage of async with with a custom object. – Dell

Thanks, I do prefer to store it in my code instead of having large async block. Does it make sense to store it as a global variable session before loop.run_forever() in my code, and then to access it from other python modules? – Asyut

@Asyut The async block doesn't need to be large if you put it at the right place - e.g. a top-most function that just calls another function. You can put it in a global variable, but I think the loop needs to be running at the time. The answer shows what I consider the recommended pattern (and faithfully answers the title of the question), but YMMV. – Dell

Is the self._session = None truly necessary? – Lavabo

@LoBellin It's not, it's mostly for clarity. (Also, it might facilitate earlier freeing of in-memory data referenced by the session.) – Dell

Answer 2

You need to close the resource after finished use. The most common way is to implement a close function.

class TestApi:
    def __init__(self) -> None:
        self.session = aiohttp.ClientSession()

    async def close(self) -> None:
        await self.session.close()

    # async defs methods from here

When you use your class, you need to call this function at the end.

api = TestApi()
# Doing important api calls
await api.close()

Better yet, you can use context managers to do this automatically.

class TestApi:
    def __init__(self) -> None:
        self.session = aiohttp.ClientSession()

    async def __aenter__(self) -> "TestApi":
        return self

    async def __aexit__(
        self,
        exc_type: Exception,
        exc_val: TracebackException,
        traceback: TracebackType,
    ) -> None:
        await self.close()

    async def close(self) -> None:
        await self.session.close()

    # async defs methods from here

Then use it like so:

async with TestApi() as api:
    # Doing important api calls
    ...

Just remember, the class has to be initialized inside a task (async function). You can read more at my answer on a similiar question at the aiohttp github repository.

Push