How to make python .post() requests to retry?

Question

I'm trying to implement requests retry in Python.
It works like charm with .get() requests, but a .post() request never retries, regardless of a status code. I'd like to use it with .post() requests.

My code:

from requests.packages.urllib3.util import Retry
from requests.adapters import HTTPAdapter
from requests import Session, exceptions

s = Session()
s.mount('http://', HTTPAdapter(max_retries=Retry(total=2, backoff_factor=1, status_forcelist=[ 500, 502, 503, 504, 521])))
r = s.get('http://httpstat.us/500')
r2 = s.post('http://httpstat.us/500')

So, the .get() requests do retry and the .post() ones do not.

What's wrong?

Vagrant

Is it supposed to work like you expect? GET requests won't harm data, but multiple POST might. I haven't read through the requests API documentation, but it sounds reasonable if this is by design. – Raylenerayless

Answer 1

In urllib3 POST is not allowed as a retried method by default (since it can cause multiple inserts). You can force it though:

Retry(total=3, allowed_methods=frozenset(['GET', 'POST']))

See https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry

Cockadoodledoo

Could you explain the logic behind using frozenset in this snippet, please? – Debarath

@AmirAfianian Check out linked documentation for Retry. The reason is to be type-consistent with default value of allowed_methods=frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE'}) – Cockadoodledoo

Note that if you want to retry "POST" but also keep all of the default methods you can do allowed_methods=frozenset([*list(Retry.DEFAULT_ALLOWED_METHODS), 'POST']) – Eustasius

Answer 2

You can use tenacity.

doc: https://tenacity.readthedocs.io/en/latest/

And you can log before or after

pip install tenacity

import logging

logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3), before=before_log(logger, logging.DEBUG))
def post_something():
    # post
    raise MyException("Fail")

Guthrie