Change request headers between subsequent retries

Question

Consider an http request using an OAuth token. The access token needs to be included in the header as bearer. However, if the token is expired, another request needs to be made to refresh the token and then try again. So the custom Retry object will look like:

s = requests.Session()
### token is added to the header here
s.headers.update(token_header)
retry = OAuthRetry(
        total=2,
        read=2,
        connect=2,
        backoff_factor=1,
        status_forcelist=[401],
        method_whitelist=frozenset(['GET', 'POST']),
        session=s
    )
adapter = HTTPAdapter(max_retries=retry)
s.mount('http://', adapter)
s.mount('https://', adapter)
r = s.post(url, data=data)

The Retry class:

class OAuthRetry(Retry):
    def increment(self, method, url, *args, **kwargs):
        # refresh the token here. This could be by getting a reference to the session or any other way.
        return super(OAuthRetry, self).increment(method, url, *args, **kwargs)

The problem is that after the token is refreshed, HTTPConnectionPool is still using the same headers to make the request after calling increment. See: https://github.com/urllib3/urllib3/blob/master/src/urllib3/connectionpool.py#L787. Although the instance of the pool is passed in increment, changing the headers there will not affect the call since it is using a local copy of the headers.

This seems like a use case that should come up frequently for the request parameters to change in between retries.

Is there a way to change the request headers in between two subsequent retries?

Rockribbed

Answer 1

No, in current version of Requests(2.18.4) and urllib3(1.22).

Retrys is finally handled by openurl in urllib3. And by trace the code of the whole function, there is not a interface to change headers between retrys.

And dynamically changing headers should not be considered as a solution. From the doc:

headers – Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers.

headers is a param passed to the function. And there is no guarantee that it will not be copy after passed. Although in current version of urllib3, openurl does not copy headers, any solution based on changing headers is considered hacky, since it's based on the implementation but not the documentation.

One work around

Interrupt a function and edit some verible it's using is very dangerous.

Instead of injecting something into urllib3, one simple solution is that check the response status and try again if needed.

r = s.post(url, data=data)
if r.status_code == 401:
    # refresh the token here.
    r = s.post(url, data=data)

Why does the original approach not work?

Requests copy the header in prepare_headers before sending it to urllib3. So urllib3 use the copy created before editing when retrying.

Silveira

This is a solution but not a scalable one. This means that every api call has to be retried like this. All of them need to be tested individually and essentially this is writing your own library to handle token refreshes in between requests. – Rockribbed