407 Authentication required - no challenge sent

Question

Update:
If you've just arrived at this question, the general gist is that I'm trying to make a HttpWebRequest via a proxy, and I'm getting a 407 from our strange proxy server. IE,Firefox,Chrome all manage to negotiate the proxy sucessfully, as do Adobe Air applications. It may be important that the Google Chrome web installer actually fails and we have to use an offline installer.

Thanks to Ian's link I've got it getting through to the next stage. It is now sending a token back to the proxy, however the 3rd stage isn't getting through, so the request with the username/password hash isn't being sent by .NET and consequently no HTML is returned.

I am using:

• IE6 user-agent
• Windows 7
• Scansafe proxy
• .NET 3.5

Here's the latest code that equates to the logs below:

HttpWebRequest request = HttpWebRequest.Create("http://www.yahoo.com") as HttpWebRequest;
IWebProxy proxy = request.Proxy;
// Print the Proxy Url to the console.
if (proxy != null)
{
    // Use the default credentials of the logged on user.
    proxy.Credentials = CredentialCache.DefaultCredentials;
}
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
request.Accept = "*/*";

HttpWebResponse response = request.GetResponse() as HttpWebResponse;
Stream stream = response.GetResponseStream();

The exception

WebException (407) Authentication Required.

The proxy being used

The proxy client is a Scansafe hardware device in our server room, which (once authenticated with NTLM) then directs your HTTP traffic to its servers to filter the traffic.

System.Net tracing output

• Here's the trace output

IE sucessfully negotiating the proxy

• Fidder output

The solution

I haven't really found a solution but thanks to Feroze and Eric I have found a workaround and discovered that the actual proxy (and not its configuration) is the main issue. It may be an obscure issue with 3 variables: .NET HttpWebRequest's implementation, Windows 7 and of course the Scansafe hardware client that sits in our rack; but without an MSDN support request I won't find out.

Thromboembolism

Answer 1

If you want to set credentials for the proxy, shouldn't you set credentials on the request.Proxy object rather than the request object?

http://msdn.microsoft.com/en-us/library/system.net.webproxy.credentials.aspx

Also, keep in mind that you need to be making a HTTP/1.1 request (or technically, any request with Keep-Alive) to successfully use NTLM/Negotiate authentication.

(Fiddler's "Auth" inspector will decompose the NTLM authentication blobs for you, if you haven't taken a look at that yet.)

Gyno

The proxy credentials are being set, I have a feeling the second part of the authentication is where the NTLM details aren't being returned, but in pastebin.com/m52afe453 it looks like the .NET client isn't even returning them, even though it's received the 2nd 407 response. – Thromboembolism

The request is HTTP 1.1 by the way - HttpWebRequest#39785641 - Request: GET http://www.yahoo.com/ HTTP/1.1 – Thromboembolism

The request is HTTP/1.1, but it's interesting that the response indicates that HTTP/1.0 semantics should be used. – Gyno

Interestingly, the proxy server is not returning the HTTP Response header "Proxy-Support: Session-Based-Authentication". That header is absolutely required for use of NTLM/Negotiate for WWW-Authentication, and I wouldn't be terribly surprised to learn that .NET requires it for Proxy-Authentication. It should be pretty simple to test with Fiddler... – Gyno

My solution has been to download CC Proxy and go via this to the proxy for now, which fixes the problem. IE is being sent the right headers back which I can't understand, and so is CC Proxy. I've updated the post so you can see what it sends back. Would there be a reason why it wouldn't send the right headers back to one client but the right ones to another? – Thromboembolism

Feroze's comments below point to the NTLM token not being sent correctly – Thromboembolism

If you actually post the .SAZ file from Fiddler, or just use the Auth tab, you can see the decomposed NTLM Blobs and see what differs between the IE case and your application case. Line #192 seem to be the key one from your System.NET log. The code calls InitializeSecurityContext with the Delegate flag, but the NTLM package doesn't actually support ISC_REQ_DELEGATE, so ISC returns SEC_E_UNSUPPORTED_FUNCTION. – Gyno

See my comment below, you have to expand the comment list to see it. As I have said already, this might be due to security updates in Vista. To investigate further, we will need an NTLM log from your client machine having the failure, along with an NTLM log of the success case with IE. That can help us debug this further. – Olibanum

I also see a slight difference in the fiddler output vs the .NET log. IE seems to be sending a cookie in the GET request. Could this have something to do with it? Maybe the proxy requires this cookie? – Olibanum

Eric, I did notice the Delegate flag being passed to ISC. However, note that when ISC was first called with Delegate flag, it worked fine and gave back the auth token to send to the server. It is just that it does not like the auth token sent back by the server. – Olibanum

Chris, is it possible to try the .NET scenario with a different username/password that the proxy supports? – Olibanum

I'll get the logs (fiddler and NTLM) tomorrow. The cookie is just from a previous visit in IE, I should've cleared the cache first. – Thromboembolism

Chris, can you clarify about the OS again? Are both client and server running Win7? Also by Win7, do you mean Windows Server 2008R2 for server (and client), or do you actually mean Windows7, i.e one of the many flavors of Win7 client? – Olibanum

(The comments here are the real answer) – Thromboembolism

Guess what I have had this problem as well - I do not have a solution but I do have a little more information. The code looks alright to me and I do not think that NTLM is sensitive to the pattern of the code. However I have found that the same code that fails on Vista/Win7/2008/ when build with VS2008 targetting .NET35 will work when compiled with VS2010 targetting .NET4. This problem only seems to affect Vista/Win7/2008 the .NET 35 and .NET 4 code works fine on XP/2003. – Bolten

Answer 2

I wrote a utility to decode the NTLM blobs that were sent in the IE and HttpWebRequest sessions.

When I look at the HttpWebRequest and IE, they both request 56bit and 128bit encryption from the server. Here is the dump of the session using HttpWebRequest

==== Type1 ----
Signature: NTLMSSP
Type: 1
Flags: E20882B7
NTLMSSP_NEGOTIATE_56
NTLMSSP_NEGOTIATE_KEY_EXCH
NTLMSSP_NEGOTIATE_128
RESERVED2
RESERVED3
RESERVED4
NTLMSSP_REQUEST_NON_NT_SESSION_KEY
NTLMSSP_TARGET_TYPE_DOMAIN
NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED
NTLMSSP_NEGOTIATE_DATAGRAM
NTLMSSP_REQUEST_TARGET
NTLM_NEGOTIATE_OEM
NTLMSSP_NEGOTIATE_UNICODE)
Domain :
Workstation:
==== Type2 ----
Signature: NTLMSSP
Type: 2
Flags: 201
NTLMSSP_NEGOTIATE_56
NTLMSSP_REQUEST_NON_NT_SESSION_KEY)
Context: D32FDDCB:63507CFA

Here is the dump from IE:

==== Type1 ----
Signature: NTLMSSP
Type: 1
Flags: A208B207
NTLMSSP_NEGOTIATE_56
NTLMSSP_NEGOTIATE_KEY_EXCH
NTLMSSP_NEGOTIATE_128
NTLMSSP_REQUEST_NON_NT_SESSION_KEY
NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY
NTLMSSP_TARGET_TYPE_SHARE
NTLMSSP_TARGET_TYPE_DOMAIN
NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED
NTLMSSP_NEGOTIATE_DATAGRAM
NTLMSSP_REQUEST_TARGET
NTLMSSP_NEGOTIATE_UNICODE)
Domain : XXXX.UK
Workstation: XXX-X31
==== Type2 ----
Signature: NTLMSSP
Type: 2
Flags: 201
NTLMSSP_NEGOTIATE_56
NTLMSSP_REQUEST_NON_NT_SESSION_KEY)
Context: D32FDDCB:63507CFA

In both IE/HttpWebRequest, they are requesting both 64 & 128bit security. However, for windows7, 128bit security for NTLM has been made the default, and without that, authentication will fail. As you can see from the server response, the server is only supporting 64bit encryption.

The following link has a discussion on a similar problem encountered by another person. http://social.msdn.microsoft.com/Forums/en-US/ncl/thread/f68e8878-53e9-4208-b589-9dbedf851198

The reason that IE works, instead of the managed app, is that IE does not actually request NTLMSSP_NEGOTIATE_SEAL | NTLMSSP_NEGOTIATE_SIGN, which end up requiring encryption. However, HttpWebRequest does request both SEAL|SIGN. This requires 128bit encryption, whereas the way IE initializes the NTLMSSP (without SEAL & SIGN), it does not require encryption. Hence IE works, whereas HttpWebRequest does not. (see the link above)

I think that if you change your security policy to allow 64bit encryption for NTLM, your managed code app will work. Or alternately, ask the proxy vendor to support 128bit encryption for NTLM.

Hope this helps.

Olibanum

I can't change the security policy unfortunately as it's domain wide for around 600 people. Thank you for the time you spent digging up the information though – Thromboembolism

Answer 3

I was having a similar issue and used the tips in the following blog post to solve the problem:

http://blogs.msdn.com/jpsanders/archive/2009/03/24/httpwebrequest-webexcepton-the-remote-server-returned-an-error-407-proxy-authentication-required.aspx

Siderosis

Thanks for the link. It now gets to Request/Response 2 as above and gets a 407 – Thromboembolism

Can you get a system.net log, and paste it here? Here are the instructions: ferozedaud.blogspot.com/2009/08/tracing-with-systemnet.html – Olibanum

Ok, I looked at the System.Net logfile that you attached. It seems as if there is an incompatibility between the Auth/security capabilities of the client machine and the proxy. Can you tell us what is the OS on the client, and the OS on the proxy? Also, what kind of proxy server are you running? – Olibanum

Also, can you condense your code down to a simple, compilable, one file, console application that demonstrates the problem? I think I might have an idea why it is not working for you, but I need to see the code first. Also, as mentioned before, I need to know the OS you are running on the client and proxy server, as well as the proxy server software. – Olibanum

Proxy server is a scansafe client (and server), and Windows 7 – Thromboembolism

Looking at the system.net log, I see that the NTLM package on the client does not like the auth-token sent by the server. When I googled, I found some links that might help understand this: technet.microsoft.com/en-us/library/dd566199(WS.10).aspx and social.msdn.microsoft.com/Forums/en-US/ncl/thread/… If you want to debug this further, you will have to enable NTLM ETL logging using instructions similar to this: blogs.msdn.com/ntdebugging/archive/2009/09/08/… – Olibanum

Or alternately, you can post your question on the NCL microsoft newsgroups (social.msdn.microsoft.com/forums/en-us/ncl) and some NCL dev can help you out. Or, you can open a support incident with microsoft PSS. – Olibanum

If you want to get the NTLM log, I can look and see if I can understand what is going on. Or otherwise, you can follow up yourself with NCL newsgroup or microsoft PSS. Good luck! – Olibanum

windows 7 ultimate. The proxy is a scansafe one, a hardware device. My Windows 7 install didn't have the inet node in the event viewer tree so I can't get the ntlm logs. The Fiddler logs had very little in the auth tab that Eric was talking about. I'm still a bit confused as to why certain applications (that don't use the .NET wrapper around wininet) don't have any problems, but this is probably creeping into the realms of an MSDN support ticket now instead of using SO. – Thromboembolism

When you say that IE is able to login, are you explicitly supplying your username/password/domain to IE, or is IE picking up the logged on user's credentials? When I look at the NTLM request difference between the IE and HttpWebRequest sessions, I see that IE is sending domain info (XXXX.UK) and WorkstationInfo, whereas Httpwebrequest is not. Can you try setting explicit credentials on the proxy, and see if that works? – Olibanum

Answer 4

Verify the following setting in secpol.msc. It fixed our issue.

Local Security Policy 
    Local Policies
        Security Options
            Network security: Minimum session security

Set to:

require 128 only for client. 

Nevillenevin

Answer 5

Can you try setting the User-Agent header on your HttpWebRequest, to the same value that IE8 is setting?

Sometimes, servers will not challenge correctly if the user-agent is not what they expect.

hope this helps.

Olibanum

Same happens regardless of the user-agent – Thromboembolism

Answer 6

Is it the way the proxy is assigned?

proxy.Credentials = CredentialCache.DefaultCredentials;

When I last used proxy with the HttpWebRequest it was assigned like this:

Assign proxy to request:

request.Proxy.Credentials = Credentials.GetProxyCredentials();

Calls method:

    public static ICredentials GetProxyCredentials()
    {
        return new NetworkCredential(AppConstants.Proxy_username, AppConstants.Proxy_password);
    }

Configure proxy in web.config

<system.net>
  <defaultProxy enabled="true">
    <proxy
      autoDetect="False"
      bypassonlocal="True"
      scriptLocation="http://www.proxy.pac"
      proxyaddress="http://proxy1.blah.com" />
  </defaultProxy>
</system.net>

Opalina

I've tried every variation of every possible property with the proxy and credentials :) It's definitely the NTLM negotiation – Thromboembolism

Answer 7

It could be related to what is in your "CredentialCache". Try this instead:

proxy.Credentials = new NetworkCredential("username", "pwd", "domain"); 

Millwork

Answer 8

How about this:

        HttpWebRequest request = HttpWebRequest.Create("http://www.yahoo.com") as HttpWebRequest;
        WebProxy proxyObject = new System.Net.WebProxy("http://10.0.0.1:8080/", true); //whatever your proxy address is

        proxyObject.Credentials = CredentialCache.DefaultCredentials;
        request.Proxy = proxyObject;

        request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
        request.Accept = "*/*";

        HttpWebResponse response = request.GetResponse() as HttpWebResponse;
        Stream stream = response.GetResponseStream();

Warrington