HTTPWebRequest.GetResponse() failing with authenticated requests through a transparent proxy

Question

We're using the HTTPWebRequest objects to make HTTP requests to our application and we're having a problem when the request requires authentication and there is a transparent proxy (Squid 3.1.10).

string url = "http://www.icode.co.uk/test/auth.php";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Credentials = new NetworkCredential("username", "password");

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);

MessageBox.Show(reader.ReadToEnd());

reader.Close();
stream.Close();
response.Close();

Our original code used the WebClient class which exhibited the same problem.

The first time this code runs, it displays the result correctly. When the code runs a second time, it fails on the GetResponse() line with:

System.Net.WebException was unhandled
  Message="The server committed a protocol violation. Section=ResponseStatusLine"
  Source="System"
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at Dummy.DummyForm.button1_Click(Object sender, EventArgs e) in H:\Trial\Dummy\DummyForm.cs:line 42
       at ...

On Windows 7, restarting the process causes it to recover and work once, but Server 2003 requires a full reboot.

Looking at the network capture, two requests are identical to start with, the initial unauthenticated request is sent and the server replies, but the failing requests sends the 2nd authenticated request in the middle of the initial reply as if it's ignoring the Content-Length header (which is correct). It then receives the rest of the initial reply and fails with the protocol error.

It does seem odd that the client (HTTPWebRequest) doesn't close the connection cleanly though.

When the proxy is not in use (non port 80 or internal traffic) the requests all work as expected. When there is no authentication, it also works as it only makes the single request.

I've already reduced the problem code to the minimum and reproduced it with the MSDN sample, but does anyone know if this is a known issue or a problem in our (.NET or Squid) configuration?

Bistoury

Not sure if this is where the issue lies, but are you making sure to close your StreamReader when you are done? – Hooray

Thanks for adding the screenshot. Did you strip Authorization information out of the first GET in it, or did it just not include any? – Hooray

I've removed my answer, as it is clear that it does not apply. – Hooray

@JonSenchyna It never does, that's normal HTTP behaviour. First send the request unauthenticated, then on the challenge, send the correct details. – Bistoury

I did not know that. Thanks for letting me know. Since the target application is yours, how is the ContentLength being set? I know I had issues on a project in the past trying to generate the correct ContentLength myself, since it is the size in number of bytes, rather than characters, of the response. – Hooray

It's "plain text" converted (local codepage) into a byte array which is then passed to the function that creates the header with the size taken from that byte array. You have prompted me to try on a different type of server though to eliminate our code entirely... – Bistoury

Hope it works out for you. Sorry I couldn't help you find the root issue iself. I think I was trying a similar approach in the past and getting issues due to the size not being the same as the size of the byte array. I don't recall what I did to get it working though. – Hooray

@JonSenchyna: Meh, exactly the same problem with a test PHP script :o( Turning off .KeepAlive also resolves it but I'd prefer not to do that for performance) – Bistoury

Can you bypass the proxy for authentication, and then when you actually need it for cached content enable traffic through at that point? – Curnin

@EkoostikMartin: It's not an explicit proxy configuration, it's transparent and configured on our gateway for all port 80 traffic. Furthermore, this is a commercial product so we have no control of the configuration of the user's network. – Bistoury

Can you post the conversation sans-proxy for comparison? – Evaluate

@Wug: Possibly tomorrow but as it's plain Connection: Close on the server's side, it is just 2 individual requests, nothing special. – Bistoury

Can you get wireshark running and get a raw dump of a transaction, both with and without the proxy? If you cant do it on the client's end, can you do it on yours? Transparent proxies sometimes take protocol shortcuts and do other things that make them less transparent than they would be ideally. It's possible that something the proxy is doing on a TCP IP level is interfering. – Evaluate

@Bistoury I second Wug's request. Would be helpful to see the working response as well as the existing broken response. The response doesn't include an encoding type in the Content-Type http header field. Can you also confirm the size of the response that is is 3722 bytes? – Vidavidal

It's split accross a couple of requests but here's a packet cpature showing the initial request (that works) followed by a subsequant request (that fails). When the proxy isn't in use, the server only supports Connection: Close so the issue doesn't show up. In all cases, the client seems to forcable close the socket (RST) rather than doing negotiated shutdown. – Bistoury

Have you tried manually adding a wwww-auhenticate header? – Darlenedarline

Answer 1

Since it only fails the second time, would

request.KeepAlive = false;

make a difference?

Hulett

It does but I'm reluctant to use that as a permanant fix as it's more of a workaround rather than a fix for the initial problem. I'll accept it anyway – Bistoury

Answer 2

I think NTLM authentication (NetworkCredential) does not work at the same time with transparent proxy feature of SQUID. :-(

http://www.squid-cache.org/mail-archive/squid-users/201110/0025.html

Could you try another authentication scheme?

Mazzard

I'm not using NTLM authentication. The NetworkCredential object is scheme agnostic. The proxy isn't authenticated at all, just the HTTP resource we're requesting, with Basic authentication. – Bistoury

Answer 3

Try authenticating yourself, with

request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(username + ":" + password));

before the request.GetResponse();

This worked for me. First I tried putting in the whole string myself, which didn't work!

Valley