Why does WebResponse never end when reading twitter firehose stream?

Question

The following function will pull down first X messages from Twitter firehose, but looks like WebResponse blocks and never exits the function:

public void GetStatusesFromStream(string username, string password, int nMessageCount)
{
    WebRequest request = WebRequest.Create("http://stream.twitter.com/1/statuses/sample.json");
    request.Credentials = new NetworkCredential(username, password);

    using (WebResponse response = request.GetResponse())
    {
        using (var stream = response.GetResponseStream())
        {
            using (var reader = new StreamReader(stream))
            {
                while (!reader.EndOfStream)
                {
                    Console.WriteLine(reader.ReadLine());

                    if (nMessageCount-- < 0)
                        break;
                }
                Console.WriteLine("Start iDispose");
            }
            Console.WriteLine("Never gets here!!!");
        }
    }

   Console.WriteLine("Done - press a key to exit");
   Console.ReadLine();
}

But the following works fine:

public void GetStatusesFromStreamOK(string username, string password, int nMessageCount)
    {
    byte[] encbuff = System.Text.Encoding.UTF8.GetBytes(username + ":" + password);
    //request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(encbuff));

    string requestString = "GET /1/statuses/sample.json HTTP/1.1\r\n";
    requestString += "Authorization: " + "Basic " + Convert.ToBase64String(encbuff) + "\r\n";
    requestString += "Host: stream.twitter.com\r\n";
    requestString += "Connection: keep-alive\r\n";
    requestString += "\r\n";

    using (TcpClient client = new TcpClient())
    {
        client.Connect("stream.twitter.com", 80);

        using (NetworkStream stream = client.GetStream())
        {
            // Send the request.
            StreamWriter writer = new StreamWriter(stream);
            writer.Write(requestString);
            writer.Flush();

            // Process the response.
            StreamReader rdr = new StreamReader(stream);

            while (!rdr.EndOfStream)
            {
                Console.WriteLine(rdr.ReadLine());
                if (nMessageCount-- < 0)
                    break;
            }

        }
    }

   Console.WriteLine("Done - press a key to exit");
   Console.ReadLine();

}

What am I doing wrong?

Tortricid

Do you know where it is hung? Can you attach a debugger and get a stack trace? Offhand, there is nothing wrong with your code. It might be that the server is misbehaving, or a bug in StreamReader. You could use techniques in ferozedaud.blogspot.com/2009/08/tracing-with-systemnet.html to figure out what is happening under the covers. That would be a good starting point. – Backstretch

If I remove the using statements and replace with reader.close() I see the same problem, and the network monitor shows the app is still receiving data but not returning from reader.close(). PS - only tested behaviour on Vista. – Tortricid

I cannot reproduce it (VS 2008 on Windows 7 64bit). It's working totally fine... – Alica

Answer 1

Cast your WebRequest as an HttpWebRequest, then before the break call request.Abort()

Nullity

Thanks, adding request.Abort() did the trick, regardless of casting the WebRequest as a HttpWebRequest. Not sure why the original code worked for some, but would strongly advise people to add request.Abort to their code if they want to run on a wide range of .NET clients. – Tortricid

I spent 12 hours pulling my hair over this :^( – Frisby

Answer 2

Looks like it has something to do with the disposal process or "using"...

The following code works fine (no "using" statements):

public static void GetStatusesFromStream(string username, string password, int nMessageCount)
    {
        WebRequest request = WebRequest.Create("http://stream.twitter.com/1/statuses/sample.json");
        request.Credentials = new NetworkCredential(username, password);

        WebResponse response = request.GetResponse();
        {
            var stream = response.GetResponseStream();
            {
                var reader = new StreamReader(stream);
                {
                    while (!reader.EndOfStream)
                    {
                        Console.WriteLine(reader.ReadLine());

                        if (nMessageCount-- < 0)
                            break;
                    }
                }
                Console.WriteLine("Never gets here!!!");
            }
        }

        Console.WriteLine("Done - press a key to exit");
        Console.ReadLine();
    }

Scriptural

Agree it gets further, but only because you're delaying the .NET garbage collection. If you run the function a couple of times the garbage collection will kick in and the program will hang as before. – Tortricid