Python HTTP request with controlled ordering of HTTP headers

Question

I am programming a client interface to a restful web service in python and unfortunately the web service requires custom headers to be present in the request. I have been using Requests for this however the web service also requires the headers to be in a specific order in the request. I haven't been able to figure out how Requests orders the headers and see if there is a way to be able to control this ordering.

I am also open to using another module other than Requests in my application if someone has a recommendation.

Cyna

If I were you I would file a bug report with the maintainers of the API as this is a clear violation of the HTTP spec. RFC 2616 Section 4.2 states: The order in which header fields with differing field names are received is not significant. Requiring a specific order for headers isn't just inconvenient, it's incorrect. – Seating

Completely agree but unfortunately that will take some time and I am under pressure to complete this project. Thanks. – Cyna

Answer 1

Updated answer

The answer below concerns versions below 2.9.2. Since version 2.9.2 (around April 2016) using an OrderedDict works again.

Old answer

It looks like it was possible some time ago using just the built-in functionality (issue 179). I think it is not anymore (issue 2057) and one of the reasons is mentioned in another comment by num1. I have used a following solution/workaround.

import requests
import collections

class SortedHTTPAdapter(requests.adapters.HTTPAdapter):
    def add_headers(self, request, **kwargs):
        request.headers = collections.OrderedDict(
            ((key, value) for key, value in sorted(request.headers.items()))
        )

session = requests.Session()
session.mount("http://", SortedHTTPAdapter())

In the example headers are just sorted but one can order them in any way. I chose that method after going through requests code and reading the method's docstring:

Add any headers needed by the connection. As of v2.0 this does nothing by default, but is left for overriding by users that subclass the HTTPAdapter <requests.adapters.HTTPAdapter> class.

For absolute control one could override the send method probably.

Fineberg

it should be enough to use OrderedDict – Homophone

I've answered in a comment to the original @michaelmeyer's answer. – Fineberg

Answer 2

You can try to use the OrderedDict class to store the headers, instead of request's default one:

>>> from collections import OrderedDict
>>> from requests import Session
>>> s = Session()
>>> s.headers
CaseInsensitiveDict({'Accept-Encoding': ... 'User-Agent': ... 'Accept': '*/*'})
>>> s.headers = OrderedDict([('User-Agent', 'foo-bar'), ('Accept', 'nothing')])
>>> s.headers
OrderedDict([('User-Agent', 'foo-bar'), ('Accept', 'nothing')])

Brashy

Unfortunately even though they are correct in s.headers if you send the request r = s.get(url) and then check the headers with r.request.headers they aren't in the same order as s.headers. Weird. – Cyna

This doesn't work because requests uses its own CaseInsensitiveDict which subclasses dict. When you pass in headers it initializes its own dict with the one you provide. – Unclothe

@num1: the docs claim it should be enough to override session headers (with OrderedDict). The tests (2 years) suggest that it should work. – Homophone

@Homophone Thanks for noticing the change. Those 2 years is quite a big approximation (I can see that GitHub shows exactly that). To be more precise, it looks like the OrderedDict came in a commit from April 2016 and is available since version 2.9.2. Previous answers are from 2013 and 2015. Maybe add another answer and note this is again possible since version 2.9.2? It could help current users. – Fineberg

s.headers = OrderedDict(..) is the current answer. Here's another code example (text in Russian). You could update your answer too if you'd like. – Homophone

If you pass an OrderedDict to the headers keyword argument, that will provide an ordering. But the ordering of the default headers used by Requests will be preferred, which means that if you override default headers in the headers keyword argument, they may appear out of order compared to other headers in that keyword argument. If this is problematic, users should consider setting the default headers on a Session object, by setting Session to a custom OrderedDict. That ordering will always be preferred. requests.kennethreitz.org/en/master/user/advanced/… – Incident

@Incident I am passing an ordered dict to session.headers, but the session.request.headers are changing the order. 'session.headers = HEADERS'; page = session.get(BASE_URL, allow_redirects = True); print(page.request.headers) – Highsmith

@Incident Thanks so much for your explanation. You managed to help me pinpoint an issue so I switched it to use Session().headers and now all good! – Save