How to get log of every retry attempt made by Python Requests Library?

Question

I'm using following code to implement retry mechanism in python requests.

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = Session()
retries = Retry(total=5,
                raise_on_redirect=True, 
                raise_on_status=True,
                backoff_factor=1,
                status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))
s.mount('https://', HTTPAdapter(max_retries=retries))
response=s.get('https://example.com')

It is working perfectly, however all this happens silently. What I want is to get informed whenever a retry attempt is made and if possible why that retry was happened. I want something like this for every retry attempt.

print(Exception.__class__)
print('Retrying after' + x + 'seconds')

Is it possible?

Contour

Answer 1

You can write your own Class to add logs it should be inherited from the Retry.

class LogRetry(Retry):
  """
     Adding extra logs before making a retry request     
  """      
  def __init__(self, *args, **kwargs):
    logger.info(f'<add your logs here>')
    super().__init__(*args, **kwargs)

retries = LogRetry(<your args>)

Carrol

Doesn't this just log with the retry object is created? It doesn't actually log the retry. – Proprioceptor

@DanDiephouse urllib3 creates new instances of the respective Retry class with the new number of retries left if a retry happens (it's implemented in a recursive fashion) github.com/urllib3/urllib3/blob/… – Inexpungible

Answer 2

Extremely hacky but it works, it won't log when a retry isn't issued.

class LogOnRetry(Retry):
  
    def __init__(self, *args, __logger: logging.Logger = None,**kwargs):
        self.__logger = kwargs.pop("__logger")
        super().__init__(*args, **kwargs)

    def new(self, **kw) -> Retry:
        kw["__logger"] = self.__logger
        return super().new(**kw)

    def increment(self, method: str | None = None, url: str | None = None, response: BaseHTTPResponse | None = None, error: Exception | None = None, _pool: ConnectionPool | None = None, _stacktrace: TracebackType | None = None) -> Retry:
        if self.__logger:
            pass
            # log here, get access to url, method and response used.
        return super().increment(method, url, response, error, _pool, _stacktrace)

Alleneallentown

Answer 3

Best option I found was to increase logging for urllib:

    'urllib3.util.retry':
    level: DEBUG
    handlers: [console]
    propagate: no

Output is:

2023-01-20 09:

59:41.018 | DEBUG    | logging:callHandlers:1706 - Starting new HTTP connection (1): x.x.x.x:8001
2023-01-20 09:59:48.263 | DEBUG    | logging:callHandlers:1706 - Incremented Retry for (url='/path/'): LogRetry(total=4, connect=None, read=None, redirect=None, status=None)
2023-01-20 09:59:48.264 | WARNING  | logging:callHandlers:1706 - Retrying (LogRetry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x28894bf10>, 'Connection to x.x.x.x timed out. (connect timeout=5)')': /path/

Proprioceptor