Getting "underlying connection was closed" on HttpWebRequest

Question

I have an application written in VB.NET (NOT asp.net, it is a Windows Console app). I am trying to call a url (an html page) and get back the response into a string. The response is straight JSON, no html tags whatsoever. It opens with { and closes with }.

I create the HttpWebRequest object fine. Then call req.GetResponse(). As soon as I do this I get the error The underlying connection was closed: The connection was closed unexpectedly. I've been googling and checking stackoverflow, and tried everything I've found that applies (a lot of it has to do with WCF service configurations, which don't apply).

Here is my code:

Public Function GetJSON(ByRef d As db.Device) As Boolean
    Try
        d.Url = "http://" & d.IpAddress & ini.doc.<svc>.<url>.Value

        Dim req As HttpWebRequest = HttpWebRequest.Create(d.Url)
        // req.Accept = "*/*"
        // req.Timeout = 30000
        // req.ReadWriteTimeout = 30000
        // req.KeepAlive = False
        // req.UseDefaultCredentials = True
        // req.CachePolicy = HttpWebRequest.DefaultCachePolicy
        // req.Proxy = HttpWebRequest.DefaultWebProxy
        // req.ProtocolVersion = New System.Version(1, 0)

        Dim rsp As HttpWebResponse = req.GetResponse()

        Return True
    Catch ex As Exception
        Log(ex.Message)
        Return False
    Finally
        rsp = Nothing
        req = Nothing
    End Try
End Function

The commented out lines (wrong comment style, but so SO will parse it right) are all the things I have tried so far based on what I've found online. None of them fixed it. I have verified that the URL being built is correct; if I call the exact same URL in my browser it gives back exactly the right expected response.

I've tried wiresharking it... and i see the actual expected, complete data come back in the shark output, and then a few lines, and then a red line which says: http > 51943 [RST] Seq=1607 Win=0 Len=0 which is the last line to show up before .NET is throwing the error.

I also tried turning on System.Net tracing/logging per a post here on SO, and in the output file from that I see similarly all the expected JSON data does come back, but after it comes back it throws these lines in the .NET trace log:

System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive()   -> 1605#1605
System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Receive()
System.Net.Sockets Verbose: 0 : [7040] Data from Socket#60467532::Receive
System.Net.Sockets Verbose: 0 : [7040] 00000000 :                                                 : 
System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive()   -> 0#0
System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Dispose()
System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816:: - The underlying connection was closed: The connection was closed unexpectedly.
System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816::GetResponse - The underlying connection was closed: The connection was closed unexpectedly.

Any ideas where to go next to try to figure this out? We are reading this data out of some environmental-monitoring sensor devices, and they gave us this url to use.

Two things that really get me and confuse me on this are that
a) it works perfectly fine when called in a browser
b) both WireShark and .NET tracing show all the data actually IS coming back, and the framework is for some reason excepting AFTER receiving all the data!

The WebException itself is being very little use, as its InnerException is null and its Status just says "ConnectionClosed {8}"

Thanks in advance!!!

UPDATE 08/18 1130: I have also tried now using just System.Net.WebRequest as opposed to HttpWebRequest. This did not make any difference either.

UPDATE 08/18 1222: I just tried switching my code instead of using [Http]Web[Request|Response] to dimming up a WebClient object instead and using its DownloadString() method. This however also throws the same exact error.

UPDATE 08/18 1230: Tried using My.Computer.Network.DownloadFile() - also gets the same connection closed error.

Lemuel

Here is a pastebin of my complete network.log .NET trace file: pastebin.com/QfY05a11 . You can see all the JSON data coming back on line 41. – Lemuel

Here is a pastebin of the Wireshark traffic log: pastebin.com/0hbEQerc . You can see the data coming back on line 607. – Lemuel

Answer 1

Can you post the entire contents of the trace log on pastebin.com and post a link here?

You might be getting this exception because the server might say in "Content-Length" header that it is sending N bytes of entity, but is actually sending less than N bytes and closing the connection.

Answer:

Thanks for the data. From the tracelog and wireshark trace, it seems that the server is not sending any response headers, and directly sending the data. This is a HTTP protocol violation. That is why the client is throwing the exception.

Mistassini

Yes, from the tracelog I can see that the server is not responding with any valid HTTP response headers. The server should first send HTTP response headers and then send the JSON payload. That is the problem – Mistassini

Also there is a bug in your code. You are not disposing the HttpWebResponse object that you get from calling GetResponse(). You need to call Dispose() on that object, otherwise you will run out of connections to the server. – Mistassini

Thanks for the info. I'll have to fix the code at the office tomorrow. I DO have a question though then (which I guess I can find out tomorrow also)... would switching to just a System.Net.WebRequest work as opposed to HttpWebRequest, seeing as it theoretically should not be looking for HTTP headers then? – Lemuel

Hello, thanks for you help thus far. I have tried using a plain WebRequest object instead of HttpWebRequest, however this did not help any. There must be some way to tell it to accept this, maybe by adding headers or something to my Request? I also went to put in code to dispose my Response object, and it does not actually have a Dispose() method, so in a finally block I just set both rsp and req to Nothing. – Lemuel

I think I didnt make myself clear. Changing your client code will not solve the actual problem. The actual problem is that the server is not well behaved - it is sending a response body without sending the response headers. You should fix the server, if you have the code for it. Using WebRequest instead of HttPWebRequest is not going to fix it. – Mistassini

ALso, if you cannot fix the server, then I would suggest that you use Socket/TcpClient directly and read the response JSON. – Mistassini

Thanks for pointing me in the direction of TcpClient... with that I was able to dig around and make it work perfect! :) . CheerZ! – Lemuel

@Mistassini - regarding the disposal of the HttpWebResponse object: Can it be disposed? Intellisense is not providing the .Dispose() method. – Hagiography

Yes, HttpWebResponse has a Dispose() method. It is inherited from WebResponse. Look it up in the docs. – Mistassini

Answer 2

This helps me. I hope it helps someone as well.

Right after you create HttpWebRequest Object, add this line.

System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;

What it means is specifies the Secure Socket Layer (SSL) 3.0 as security protocol

http://support.microsoft.com/kb/915599

Setscrew

Answer 3

Here is how i got it working... Thanks to feroze for pointing me in the right direction!
(points awarded)

Public Function GetJSON(ByRef d As db.Device) As Boolean
    Try
        Dim tcp = New TcpClient()
        tcp.Connect(d.IpAddress, 80)

        Dim ns = tcp.GetStream()

        Dim req As Byte() = System.Text.Encoding.ASCII.GetBytes(
            "GET /getdata.htm HTTP/1.1" & vbCrLf & vbCrLf
        )
        ns.Write(req, 0, req.Length)

        Dim rsp(2048) As Byte, rcv As Integer
        Do
            rcv = ns.Read(rsp, 0, rsp.Length)
            d.JSON &= System.Text.Encoding.ASCII.GetString(rsp, 0, rcv)
        Loop Until rcv = 0

        tcp.Close()

        Return True
    Catch ex As Exception
        Log(ex.Message)
        Return False
    End Try
End Function

Lemuel

tcp.Close() should be called from inside "finally", otherwise it won't be called if there's an exception before the .Close() call. – Lorie

Answer 4

Check the latency of the request. If it is more than a few ms then it is worth disabling the Nagle Algorithm

 ServicePointManager.UseNagleAlgorithm = false;

Turkish

Answer 5

I was having the same issue regarding a request WITHOUT a body. In my case, setting the ContentLength to ZERO fixed the problem.

Exhibitionist

Answer 6

In my case the problem was we used "http://.." as service address instead of "https://"...

Adieu

Answer 7

I have also received this error when a required header wasn't present

Jermayne

which header was it in your case? – Equilibrist

I was missing a custom header for an API developed by another team. I assumed the header was optional since they had a default value defined. – Jermayne