Strict ordering of HTTP headers in HttpWebrequest

Question

In spite of the RFC stating that the order of uniquely-named headers shouldn't matter, the website I'm sending this request to does implement a check on the order of headers.

This works:

GET https://www.thewebsite.com HTTP/1.1
Host: www.thewebsite.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 etc

This doesn't work:

GET https://www.thewebsite.com HTTP/1.1
Accept: */*
User-Agent: Mozilla/5.0 etc
Host: www.thewebsite.com
Connection: keep-alive

The default HttpWebRequest seems to put the Host and Connection headers at the end, before the blank line, rather than just after the url.

Is there any way (using a fork of HttpWebRequest or some other library in Nuget even) to specify the order of headers in a HttpWebRequest?

If possible, I'd rather not start going down the route of implementing a proxy to sort them or having to code the whole thing up using a TcpClient.

I'd appreciate any hints at all on this.

Update: With Fiddler running, header order in HttpWebrequest can be re-shuffled in CustomRules.cs. Still no closer to a solution without a proxy though.

Waldrup

Please elaborate what does it mean This doesn't work. What http status code does the server return? – Insistent

It returns a 403 Forbidden. – Waldrup

Did you try to perform these requests from Postman? – Insistent

Quite frankly I don't think so there is any third party library that supports header sorting. I've checked RestSharp's and Flurl's solution for request headers and none of them are ordered. – Insistent

I tried Postman and checked what was being sent with Wireshark. You can't order the headers with Postman, so it fails each time with a 403. Fiddler is a better tool for this. – Waldrup

Can you update the question with how the web request was created in C#? – Piave

in regards to "I'd rather not start going down the route of implementing a proxy to sort them"... I think you should go down the route of implementing a reverse proxy. I think working with a site that is stricter than the RFC standard merits that layer of abstraction. consider the scenario where order is changed\different in a staging environment, or varies by what resource you're requesting – Sportive

I think you will effectively require a rule\policy based feature in place for that uncertainty. – Sportive

excuse me, I meant forward proxy for outbound requests – Sportive

Is this for .NET Framework? Or .NETCore? – Quinn

.NET Framework. – Waldrup

It turns out my solution didn't work on .net Framework. I'm starting to think that this approach to reordering headers (also used by @tontonsevilla) could potentiality break in a future update to .net Core. Nowhere in the documentation does it say that we at allowed to do this, it's just exploiting a side effect due to the internal collection used. – Breathed

Answer 1

.Net Core

If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:

using System.Net;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            var request = WebRequest.Create("http://www.google.com");
            request.Headers.Add("Host", "www.google.com");
            // this will be set within GetResponse.
            request.Headers.Add("Connection", "");
            request.Headers.Add("Accept", "*/*");
            request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
            request.GetResponse();
        }
    }
}

Here is an example with HttpClient:

using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp3
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var client = new HttpClient();
            client.DefaultRequestHeaders.Add("Host", "www.google.com");
            client.DefaultRequestHeaders.Add("Connection", "keep-alive");
            client.DefaultRequestHeaders.Add("Accept", "*/*");
            client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
            await client.GetAsync("http://www.google.com");
            await client.PostAsync("http://www.google.com", new StringContent(""));
        }
    }
}

Edit The above code did not work on .Net Framework only .Net Core

.Net Framework

On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.

One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.

Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize. See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079

So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.

WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES

If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework

using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            // WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
            // If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
            var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
            var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
            var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
            {
                ["Host"] = "www.google.com",
                ["Connection"] = "keep-alive",
                ["Accept"] = "*/*",
                ["User-Agent"] = "Mozilla/5.0 etc"
            });
            field.SetValue(request, headers);
            request.GetResponse();
        }
    }

    internal class CustomWebHeaderCollection : WebHeaderCollection
    {
        private readonly Dictionary<string, string> _customHeaders;

        public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
        {
            _customHeaders = customHeaders;
        }

        public override string ToString()
        {
            // Could call base.ToString() split on Newline and sort as needed

            var lines = _customHeaders
                .Select(kvp => $"{kvp.Key}: {kvp.Value}")
                // These two new lines are needed after the HTTP header
                .Concat(new [] { string.Empty, string.Empty });

            var headers = string.Join("\r\n", lines);

            return headers;
        }
    }
}

Breathed

I also looked into having a custom GetEnumerator with a sort function, implemented on a custom WebHeaderCollection, but it required reflection to set the field as the setter within HttpWebRequest copies content into a new WebHeaderCollection. – Breathed

The WebRequest code above won't run in .NET Framework. Is this for .NET Core? – Waldrup

@Waldrup Aaah, yes I tested this on .net core. I just had a quick test and I have a solution that uses reflection. But it's not pretty – Breathed

@Waldrup I have added the ugly code. It can work and it could be cleaned up by passing a dictionary into the custom headers or making a sort function in ToString, but it is definitely error prone if the framework were to change it's implementation. – Breathed

Answer 2

Some server implement header ordering as a precaution for any attacks or spam, an article explaining Why ordering HTTP headers is important.

But the standard is, the order in which header fields with differing field names are received is not significant.

HttpWebRequest, there is no easy way to order the headers and the Connection and Host is added internally.

If ordering is really important, use the HttpClient instead, it can easily arrange the Headers based on the example of @Jason.

If you will be using HttpClient, you can create a custom HttpClientHandler and you can arrange your header from there. It can be something like this.

HANDLER

public class CustomHttpClientHandler : HttpClientHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Clear();

        request.Headers.Add("Host", $"{request.RequestUri.Authority}");
        request.Headers.Add("Connection", "keep-alive");
        request.Headers.Add("Accept", "*/*");
        request.Headers.Add("User-Agent", "Mozilla/5.0 etc");

        return await base.SendAsync(request, cancellationToken);
    }
}

IMPLEMENTATION

HttpClient clientRequest = new HttpClient(new CustomHttpClientHandler());
await clientRequest.GetAsync(url);

Estipulate

I think this is a good answer - though not complete. you could improve this answer by providing instructions on how to reuse this solution. That is, since OP, is looking for non proxy or project coded solution, it stands to reason that this behavior would be a package and framework integration, and that it may require\support some configuration efforts. – Sportive

@Estipulate it turns out my solution didn't work on .net Framework. I'm starting to think that this approach to reordering headers could potentiality break in a future update to .net Core. Nowhere in the documentation does it say that we at allowed to do this, it's just exploiting a side effect due to the internal collection used. – Breathed