Accept-Encoding headers being sent by browser but not received by server

Question

I have been trying to debug this for weeks. All of the browsers on all of the clients on my home network are sending 'Accept-Encoding: gzip,deflate'. However, that header is somehow, somewhere being dropped before the request makes it to a web server. For example, http://www.whatsmyip.org/http_compression/ says 'No, your browser is not requesting compressed content'.

I've used Fiddler to make sure that all of my browsers are indeed sending the header. I've swapped out my router. I've turned off all anti-virus software.

Brighthouse/Roadrunner (the local cable ISP) says they are not doing any filtering (and I can't see why they would in this case).

Any suggestions would be most welcome!

Hobnailed

That's pretty weird. If it were happening to me, and one of the machines involved were a laptop, I'd take it over to my neighborhood coffee shop and see what happens from there. It might help if you could post all the headers that you see in outgoing HTTP requests. – Gorham

For the sake of completeness, I will try it at the coffee shop with my laptop later. Here are the headers I am sending: <code>Host www.whatsthatbug.com User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) Accept text/css,/;q=0.1 Accept-Language en-us,en;q=0.5 Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 115 Connection keep-alive</code> but the server is not seeing 'Accept-Encoding' – Hobnailed

Answer 1

Try it with HTTPS.

If you are browsing a site via HTTPS, nothing between your browser and the web server can alter any HTTP-level aspect of the the request or response, including whether compression is enabled, without you having immediate and clear knowledge of that fact (check the site's certificate in your browser address bar and see if it's legit).

Suppletory

Ok, I changed the logs on my Apache server to log Accept-Encoding headers. And what shows up in the logs is "-" for that field. So somewhere between my router and the server (any server, not just mine) the Accept-Encoding header is being stripped. Now I'm really confused! – Hobnailed

If your Apache server is responsible for SSL and has a certificate, if you try this with SSL, you will be able to see whether or not there is a "Man in the Middle" changing the request on the fly. With SSL, your browser can detect and warn you about any "Man in the Middle" fiddling with the request. – Suppletory

So if it's being generated by my browser, it's not being changed on my network, it's not being changed mid-stream, and it's not being seen on any remote server, where could the Accept-Encoding header be going? – Hobnailed

Odd.... What about e.g. using Firefox with Firebug (Net panel) and browsing to any other site over SSL, such as https://twitter.com/? What does the response's Encoding: header say? – Suppletory

Accept-Encoding gzip,deflate was indeed in Firebug's results for twitter.com. But Firebug is just showing the headers I'm sending, not necessarily what the remote site receives, right? Because if I go to whatismyip.org/http_compression, for example, it tells me I am not requesting compression. Somewhere the Accept-Encoding header is disappearing still. – Hobnailed

Firebug will show both the headers the browser sends to the server as well as the headers received by the browser from the server. If the server compresses the response body, then it must also add a header Encoding: gzip. – Suppletory

Right. And the response headers do not have Encoding: gzip. That's what I'm saying :-) I'm sending the Accept-Encoding header from each of the browsers on my network. None of the servers are receiving that header if I use http, but they are if i use https. – Hobnailed

Nice. The implication is that you have a Man in the Middle modifying your HTTP requests in-transit and stripping out the Accept-Encoding header. The question now is to narrow down who that Man in the Middle is. I doubt it's the router doing that, especially if you checked with two different router models. Perhaps you can double-check with the ISP, or the ISP's upstream? – Suppletory

I'll need to try escalating it to get anywhere with the ISP - the first response I got was less than helpful :-) This is really strange. I appreciate you trying to help. – Hobnailed

Answer 2

I had the Accept-Xncoding issue and determined it was CA Internet Security Suite causing the issue. Disabling wan't enought, you had to uninstall and then clear IE Cache.

Slide

Answer 3

Check your antivirus software. It's probably intercepting your outbound traffic and modifying headers on the fly in order to get uncompressed content. Lazy programmers don't like to include decompression methods themselves, or deal with chunked encoding.

Norton Internet Security will overwrite accept encoding with this line:

---------------: ----- -------

McAfee overwrites with this:

X-McProxyFilter: *************

something I haven't identified yet overwrites with this:

Accept-Xncoding: gzip, deflate

You are probably in the same boat. I read Zone Alarm wipes out the encoding header entirely (which means recalculating the size of the packet, but why should they care how much load they introduce on your system?). If you're running Zone Alarm, turn off the 'internet privacy option' or whatever it is, and try again.

Everytime I've seen this problem, it has been the result of shitty antivirus. Completely disabling someone's ability to receive compressed content without letting them know is dirty.

Tighe