HttpWebRequest stops working suddenly, No response received after few requests

Question

I’m working with a WPF .net 4.0 Application. I have a search bar. For each search token I need to do 8 http request to 8 separate URLs to get search results. I send 8 requests to server after 400 milliseconds once the user stops typing in search bar. Searching for 6 to 7 search-tokens results comes very nicely. But after that suddenly HttpWebRequest stops working silently. No exception was thrown, no response was received. I'm working with Windows 7, I disabled the firewall too. I don't know where the subsequent http requests are lost.

Can anyone show me lights to fix this issue?

Below is my code for HttpWebRequest call.

public static void SendReq(string url)
{
    // Create a new HttpWebRequest object.
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

    request.ContentType = "application/x-www-form-urlencoded";
    request.Proxy = new WebProxy("192.168.1.1", 8000);

    // Set the Method property to 'POST' to post data to the URI.
    request.Method = "POST";

    // start the asynchronous operation
    request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request);

}

private static void GetRequestStreamCallback(IAsyncResult asynchronousResult)
{
    HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;

    // End the operation
    Stream postStream = request.EndGetRequestStream(asynchronousResult);

    string postData = this.PostData;

    // Convert the string into a byte array. 
    byte[] byteArray = Encoding.UTF8.GetBytes(postData);

    // Write to the request stream.
    postStream.Write(byteArray, 0, byteArray.Length);
    postStream.Close();

    // Start the asynchronous operation to get the response
    request.BeginGetResponse(new AsyncCallback(GetResponseCallback), request);
}

private static void GetResponseCallback(IAsyncResult asynchronousResult)
{
    HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;

    // End the operation
    using(HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
    {
        using(Stream streamResponse = response.GetResponseStream())
        {
             using(StreamReader streamRead = new StreamReader(streamResponse))
             {
                 string responseString = streamRead.ReadToEnd();
                 Debug.WriteLine(responseString);
             }
        }
    }
}

Dimitrovo

How certain are you that no exceptions are being thrown for the other requests? Your Close calls aren't in using statements, which means if there are exceptions, you'll be leaving response connections open, which could deadlock later requests... – Bakki

No exception is thrown. Application is completely silent. No response received GetResponseCallback is never being called after 6 to 7 requests. I have updated the code. Added using but still same issue. – Dimitrovo

Have you looked at what's going on at the network level with something like Wireshark? – Bakki

What does it mean that you read Console input in GetRequestStreamCallback? Are you really entering something before sending a request? – Leathery

@Clemens, Yes I'm posting very little amount of data too. I have few post parameters which need to send to server. I have a wrapper class of this code which supplies postData to this code. – Dimitrovo

Doesn't the console input block in GetRequestStreamCallback? – Leathery

@Clemens, No... I updated the code. please have a look. – Dimitrovo

@JonSkeet, I didn't try Wireshark but I have tried Fiddler, Fiddler is able to catch few request at the beginning. But later on Fiddler is also not receiving any requests being proxy. That means something is blocking my requests. – Dimitrovo

@Somnath: That's what I'd expect if the responses weren't being closed properly. It's odd that it's happening when you are closing them. Hmm. Can you easily test a synchronous version? – Bakki

@JonSkeet, right its a good idea to test synchronous version. I will try that soon. – Dimitrovo

Answer 1

I think i am very late, but i still want to answer your question, may be it can be helpful to others. By default HTTP Request you make are HTTP 1.1 requests. And HTTP 1.1 Request by default has Keep-Alive connection. so when you make too many request to same server .net framework only make x no. of request.

you should close all your response by response.Close()

you can also specify how many simultaneous requests you can make.

ServicePointManager.DefaultConnectionLimit = 20;

Note that you have to set DefaultConnectionLimit before the first request you make. you can find more information here on msdn.

Trenchant

Answer 2

All i can see is that in GetRequestStreamCallback you should replace

postStream.Write(byteArray, 0, postData.Length);

by

postStream.Write(byteArray, 0, byteArray.Length);

since these length aren't necessarily equal.

Leathery

Thanks Clemens, you sounds right. Just now I did the changes in my code. Even it does not solve this problem of silent failure of HttpWebRequest. – Dimitrovo

Answer 3

@Somnath I am not sure if you found this answer, but if anyone else stumbles across this post with the same issue that Somnath and I were having.

We all try to do our due diligence in keeping memory clean and clear, but with streams we always will save unexplained issues if we make sure to flush the stream before we close it.

Replace This :

postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();

With This :

postStream.Write(byteArray, 0, byteArray.Length);
postStream.Flush();
postStream.Close();

Causation

Hi Jdunn, Just now I tested your suggestion but failed. I flushed and closed all streams. So that there is no memory leak in my code. Still no luck. Please note that when I keep on typing in search-box I'm sending too may requests before some requests comes back with response. I'm just ignoring the previous responses. Now I guess I need to do something with request cancellation. Whats in your mind? Any guess..! – Dimitrovo

Answer 4

I followed all suggestion provide by all of you but couldn't stop the silent failure of HTTP request. But I found a workaround. Even myself could not reach to a final conclusion till now.

But my workaround is working well as off now without any failure.

In SendReq(string url) function i have added the following lines of code

System.Net.ServicePointManager.DefaultConnectionLimit = 100; // Just selected a random number for testing greater than 2
System.Net.ServicePointManager.SetTcpKeepAlive(true, 30, 30); // 30 is based on my server i'm hitting
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

request.KeepAlive = true;

Dimitrovo