Python urllib3: close idle connection after some time

Question

Is there a way to tell Python urllib3 to not reuse idle connections after some period of time, and instead to close them?

Looking in https://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool doesn't seem to show anything relevant.

Eirena

Answer 1

Remember:

A connection pool is a cache of database connections maintained so that the connections can be "reused" when future requests to the database are required.

You can do this is many ways (I guess):

• Set retries to one.

This breaks your connection if it failed one time. To set it:

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=1) # is zero for default
s.mount('http://', a)

---

• Change pool connections.

The "pool_connections" is the number of host-pools to keep around. For example, if you're connecting to 100 different hosts, and pool_connections=10, then only the latest 10 hosts' connections will be re-used. To set that:

s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1))
s.get('https://www.example.com')

This will stop the reuse of pools.

---

• Pool Maxsize

This is cared only if you use Session in a multithreaded environment. To set it:

s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))

---

• Configure Maxsize

he :class:~connectionpool.ConnectionPool class keeps a pool of individual :class:~connection.HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. To set it (it is, by default):

from urllib3 import HTTPConnectionPool
pool = HTTPConnectionPool('www.example.com', maxsize=0) #likely to slow you down cuz it never stores the pools

maxsize – Number of connections to save that can be reused. More than 1 is useful in multithreaded situations.

---

• Let the Pool Manager do that!

The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager num_pools to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually. So to do that:

from urllib3 import PoolManager
manager = PoolManager(1) # not the manager cleans up pools used for one time
r = manager.request('GET', 'http://www.example.com/')

Also, the docs says that:

Cleanup of stale pools does not happen immediately.

So for that Use RecentlyUsedContainer (Docs contains only one line).

Note:

Setting arguments if PoolManager affects all the pools connected thereby.

---

Hopes this helps you. Get the advanced usage docs HERE .

Nev

when connecting to aws nat gateway , the nat gateway has a fixed 350 seconds idle timeout. When urllib tries to reuse the connection it will get an explicit RST (connection reset by peer). Even if it retries it will get another connection from the pool that is also going to be RST immediately (because it was idle too). Is there anyway to tell urllib to discard proactively all connections that have been idle more that 350 seconds? – Sadirah

Answer 2

Firstly, you could define the num_pools variable so that if it crosses the value of the pools already created, then it would discard the least recently used. You could also set timeout on the connection so that the connections could automatically close after the timeout. I was also not able to find anything to determine the idle connections, but I guess this might help

https://laike9m.com/blog/requests-secret-pool_connections-and-pool_maxsize,89/

Revkah