Reading data from an open HTTP stream

Question

I am trying to use the .NET WebRequest/WebResponse classes to access the Twitter streaming API here "http://stream.twitter.com/spritzer.json".

I need to be able to open the connection and read data incrementally from the open connection.

Currently, when I call WebRequest.GetResponse method, it blocks until the entire response is downloaded. I know there is a BeginGetResponse method, but this will just do the same thing on a background thread. I need to get access to the response stream while the download is still happening. This just does not seem possible to me with these classes.

There is a specific comment about this in the Twitter documentation:

"Please note that some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example."

They point to the Appache HttpClient, but that doesn't help much because I need to use .NET.

Any ideas whether this is possible with WebRequest/WebResponse, or do I have to go for lower level networking classes? Maybe there are other libraries that will allow me to do this?

Thx Allen

Reiko

Answer 1

I ended up using a TcpClient, which works fine. Would still be interested to know if this is possible with WebRequest/WebResponse though. Here is my code in case anybody is interested:

using (TcpClient client = new TcpClient())
{

    string requestString = "GET /spritzer.json HTTP/1.1\r\n";
    requestString += "Authorization: " + token + "\r\n";
    requestString += "Host: stream.twitter.com\r\n";
    requestString += "Connection: keep-alive\r\n";
    requestString += "\r\n";

    client.Connect("stream.twitter.com", 80);

    using (NetworkStream stream = client.GetStream())
    {
        // Send the request.
        StreamWriter writer = new StreamWriter(stream);
        writer.Write(requestString);
        writer.Flush();

        // Process the response.
        StreamReader rdr = new StreamReader(stream);

        while (!rdr.EndOfStream)
        {
            Console.WriteLine(rdr.ReadLine());
        }
    }
}

Reiko

Yes, HTTP is line-oriented protocol. You need to separate commands by \r\n. – Certification

Is the stream returned by client.GetStream() actually streamed from the network stack or is it just a cached memory stream? – Ultraconservative

Answer 2

BeginGetResponse is the method you need. It allows you to read the response stream incrementally:

class Program
{
    static void Main(string[] args)
    {
        WebRequest request = WebRequest.Create("http://stream.twitter.com/spritzer.json");
        request.Credentials = new NetworkCredential("username", "password");
        request.BeginGetResponse(ar => 
        {
            var req = (WebRequest)ar.AsyncState;
            // TODO: Add exception handling: EndGetResponse could throw
            using (var response = req.EndGetResponse(ar))
            using (var reader = new StreamReader(response.GetResponseStream()))
            {
                // This loop goes as long as twitter is streaming
                while (!reader.EndOfStream)
                {
                    Console.WriteLine(reader.ReadLine());
                }
            }
        }, request);

        // Press Enter to stop program
        Console.ReadLine();
    }
}

Or if you feel more comfortable with WebClient (I personnally prefer it over WebRequest):

using (var client = new WebClient())
{
    client.Credentials = new NetworkCredential("username", "password");
    client.OpenReadCompleted += (sender, e) =>
    {
        using (var reader = new StreamReader(e.Result))
        {
            while (!reader.EndOfStream)
            {
                Console.WriteLine(reader.ReadLine());
            }
        }
    };
    client.OpenReadAsync(new Uri("http://stream.twitter.com/spritzer.json"));
}
Console.ReadLine();

Apospory

I didn't try this because I assumed the req.EndGetResponse would block in exactly the same way req.GetResponse does, leaving me with the same problem. I will give it a shot. – Reiko

I tried this without any luck. The stream returned by .GetResponseStream() is a MemoryStream containing the entire contents of the response. – Rajewski

@Darin Dimitrov, How do you deal with timeouts if there are long periods of time between data coming through on the stream? – Botch

Thanks the WebClient example worked for me when connecting to Twitter filtered stream endpoint https://api.twitter.com/2/tweets/search/stream. I only needed to replace the Credentials line with client.Headers[HttpRequestHeader.Authorization] = $"Bearer {settings.Twitter.BearerToken}"; and of course the Uri part. – Androsphinx

Answer 3

I think the modern way to do this is here:

var client = new HttpClient();
using var stream = await client.GetStreamAsync("http://stream.twitter.com/spritzer.json");

Distribution

Answer 4

Have you tried WebRequest.BeginGetRequestStream() ?

Or something like this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create (http://www.twitter.com );
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream()); 

string str = reader.ReadLine();
while(str != null)
{
   Console.WriteLine(str);
   str = reader.ReadLine();
}

Vampirism

I can't use WebRequest.BeginGetRequestStream because I am not trying to write data asynchronously to the server. Also, in your example you call GetResponse. It is at this point that the WebRequest is blocking while it downloads the server response. But the connection never closes because it is an effectively endless stream of data. – Reiko