Implementing retry for requests in Python

Question

How do I implement a retry count of 5 times, 10 seconds apart when sending a POST request using the requests package. I have found plenty of examples for GET requests, just not post.

This is what I am working with at the moment, sometimes I get a 503 error. I just need to implement a retry if I get a bad response HTTP code.

for x in final_payload:
    post_response = requests.post(url=endpoint, data=json.dumps(x), headers=headers)

#Email me the error
if str(post_response.status_code) not in ["201","200"]:
        email(str(post_response.status_code))

Whizbang

Possible duplicate of What is the best way to repeatedly execute a function every x seconds in Python? – Matson

Retries have nothing specifically to do with GET or POST, so you should easily be able to adapt the examples you found. – Badr

I've used this in the past for GET requests from an external API pypi.python.org/pypi/retrying – Ambrosius

Answer 1

you can use urllib3.util.retry module in combination with requests to have something as follow:

from urllib3.util.retry import Retry
import requests
from requests.adapters import HTTPAdapter

def retry_session(retries, session=None, backoff_factor=0.3):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        method_whitelist=False,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Usage:

session = retry_session(retries=5)
session.post(url=endpoint, data=json.dumps(x), headers=headers)

NB: You can also inherit from Retry class and customize the retry behavior and retry intervals.

Outofdoor

@DhiaTN - I'm a little confused about this. Assuming this I do res = session.get(... - will res return immediately in any case? or "wait" until there is either and error or a proper response? What happens if the maximum amount of retries is exceeded? – Eject

you will response anyway, then you have to check the status code to understand if it failed or not – Outofdoor

For me doing the above didn’t work. I used from requests.packages.urllib3.util.retry import Retry instead of from urllib3.util.retry import Retry. – Eject

This gives me an error ValueError: Unable to determine whether fp is closed. – Lindsey

Answer 2

I found that the default behaviour of Retries did not apply to POST. To do so required the addition of method_whitelist, e.g. below:

def retry_session(retries=5):
    session = Session()
    retries = Retry(total=retries,
                backoff_factor=0.1,
                status_forcelist=[500, 502, 503, 504],
                method_whitelist=frozenset(['GET', 'POST']))

    session.mount('https://', HTTPAdapter(max_retries=retries))
    session.mount('http://', HTTPAdapter(max_retries=retries))

    return session

Farlay

You should also be able to do method_whitelist=False to allow retry on POST. – Northcliffe

Is there a way I can add custom logging for each retry? – Wilmot