What does "Cannot write to closing transport" mean?

Question

I get the exception "Cannot write to closing transport" raised from aiohttp.http_writer.StreamWriter#_write, but only in a fraction of cases.

The relevant snippet.

    session: aiohttp.ClientSession
    async with session.get(url, timeout=60) as response:
        txt = await response.text()
        response.close()
        return txt

What is going on? I don't think the server-size is closing the socket.

Daughter

How is that possible? In this code, we await response.text(). So if we got to response.close(), the full HTTP response is finished, right? – Daughter

You can reproduce this: add asyncio.sleep(10) to you handler, call it and stop waiting (break connection from client side) before it respond you. You will see same error in server logs – Terrain

I think when you closing connection server still responding smth. Why do you closing connection (response.close()) manually? AFAIK connection will close automatically when __aexit__() of ClientSessionContextManager will be called. – Terrain

Smal remark: _SessionRequestContextManager not ClientSessionContextManager – Terrain

Answer 1

Answer: We should create a new session on each request. There is also no need to close the response() explicitly, as the context manager handles that.

async with aiohttp.ClientSession().get(url, timeout=60) as response:
    txt = await response.text()
    return txt

Daughter

Dedicated session for each request is what helped me fix the Cannot write to closing transport exception, although the documentation of aiohttp says the opposite: Create the session first, use the instance for performing HTTP requests and initiating WebSocket connections. (docs.aiohttp.org/en/v3.0.1/client_advanced.html#client-session). I'm using aiohttp 3.6.2 (2019-10-09) and Python 3.6.9 on Ubuntu 18.04.4 LTS. – Sargasso

As others have noted, this is exactly what the documentation advises you not to do. ClientSession() maintains a per-session pool of connection handlers and each time you create a new one and throw it away, that connection pool gets set up and torn down. It's intended to be reused. The documentation is very unhelpful on how to reuse it though! – Elaterin

Answer 2

It means that your connection already closed. It occurs when client break connection, but server still tried respond him.

Remove response.close() from your code.

Terrain

How is that possible that the HTTP response is continuing, given that response.text() has consumed all the response? – Daughter

@JoshuaFox It occurs because you close connection manually (response.close()). Don't do this. It will be closed automatically when you get out from _SessionRequestContextManager – Terrain

OK, that makes sense – Daughter

The mistake is simply that I use response.close(). If I omit response.close(), the session context-manager closes the response by itself. – Daughter

I don't use response close. But I get this error when the client disconnects sometimes. How can I handle it? Try catch on websocket emit doesnt work. – Afterbirth

@TheFool AFAIK you can't catch this. – Terrain

@TheFool or try to wrap your cs.request context manager with try ... catch – Terrain

Answer 3

This happens if a client prematurely disconnects before reading some or all of the response. You may encounter that case frequently if you're either dealing with mobile clients (which may switch between WiFi and mobile networks) or if you have views that take some time, but clients have a lower timeout. Since you can't control how clients talk to your service, it's probably safe to ignore this.

aiohttp 3.6.0 introduces code to slience this exception

Aday

Answer 4

I think the problem here is that you've already used the ClientSession object in an async context manager. When the async with aiohttp.ClientSession() as session: block exits, it tears down the session's connection pool.

Advice from others to use a single ClientSession() object per request goes directly against what the documentation says. To take advantage of the connection pooling, the session should be kept for a long time. The documentation is very unclear on how to do this though.

One way is with the asyncio_loop_local package (see here). With this package, you can do this:

_ClientSession = asyncio_loop_local.sticky_singleton_acm(aiohttp.ClientSession)

async with _ClientSession() as session:
   ... do your request ...

async with _ClientSession() as session:
   ... and another request ...

This will transparently re-use ClientSession() objects, with one per event loop and with the context managers managed cleanly for you.

Elaterin