How to read the response stream before the Http response completes

Question

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

Leonerd

GetResponse() or the callback you provide for BeginGetResponse() are called as soon as all the response headers are read, but the entire response will not be read unless it is really small or you read it. – Uball

Not clear if the OP has indeed tested any of the suggested solutions and faced any specific problems. In my experience, get response stream only gets the stream and as you read from that stream, the response is downloaded, unless of course it is a small chunk! – Stinker

Answer 1

I think this is very close to what @Zachary suggests. And it (seems to) work(s); actually I think using using as @Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.

In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.

You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?

The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.

namespace StreamReadWebRequest
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Net;
    using System.IO;

    class Program
    {
        static void Main(string[] args)
        {
            HttpWebRequest req;
            HttpWebResponse res = null;

            try
            {
                req = (HttpWebRequest)WebRequest.Create(
                        "http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
                res = (HttpWebResponse)req.GetResponse();
                Stream stream = res.GetResponseStream();

                byte[] data = new byte[4096];
                int read;
                while ((read = stream.Read(data, 0, data.Length)) > 0)
                {
                    Process(data, read);
                }
            }
            finally
            {
                if (res != null)
                    res.Close();
            }
            Console.In.Read();
        }

        private static void Process(byte[] data, int read)
        {
            Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
        }
    }
}

Vitellin

Ditto ... what scherand said :) – Tsushima

Answer 2

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:

HttpWebResponse resp = (HttpWebResponse)request.GetResponse();

As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!

However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

Kushner

This was exactly my issue. Once I closed Fiddler GetResponse() behaved as expected. Thanks! – Rus

I know this is an old thread, but adding my 2 cents. Fiddler is actually a proxy server, so it actually gets the entire response from the original server and then does what you are expecting it to do. That's why its blocking. – Nichollenicholls

Answer 3

If you set the buffer size on your read, you can read in the data in chunks... example...

 // Get the response stream
 using(Stream resStream = response.GetResponseStream())
 {

        string parseString = null;
        int    count      = 0;

        do
        {
            // Read a chunk of data
            count = resStream.Read(buf, 0, buf.Length);

            if (count != 0)
            {
                // Convert to ASCII
                parseString = Encoding.ASCII.GetString(buf, 0, count);

                // Append string to results
                sb.Append(tempString);
            }
        }
        while (count > 0);

}

Feodor

This is after the response is complete, I want to be able to read the response stream before that. – Leonerd

Answer 4

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.

After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.

If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?

Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.

Politicking

Answer 5

To elaborate a little more on what has already been said about using the web response stream to read the data, I would like to add a few remarks.

When reading data from a response stream object, it is crucial to use only that part of the buffer filled with actual data received from the server and discard parts of the buffer that remain empty with bytes of 0. The Read method of the response stream reads data sent by the server and in the case of the code shown below puts the data into the buffer byte array. Not all parts of the buffer byte array are filled with data. The Read method fills the buffer starting from index 0. In the code below the actual number of the bytes read is assigned to variable rdl.

int buffSize = 1000; 
byte[] buffer = new byte[buffSize];
int rdl = rspStrm.Read(buffer, 0, buffSize);

It is crucial to extract data from the buffer from index 0 to index rdl-1. The rest of the buffer is empty and is filled with 0 or 0x00 or NULL. If all the buffer is used, the data will be corrupted. The following code uses "System.Buffer.BlockCopy" to extract bytes from the buffer and put them into the output byte array at specific indices.

public List<byte> SendWebRequest()
{
    byte[] allBytes = Array.Empty<byte>();
    string url = "http://localhost/xmlfile.xml";
    System.Net.WebRequest request = System.Net.WebRequest.Create(url);
    if (request is not null)
    {
        int reqLg = 0;
        request.Method = "POST";
        byte[] requestContent = System.Text.UTF8Encoding.UTF8.GetBytes(" data sent by request stream");
        reqLg = requestContent.Length;
        request.ContentLength = reqLg;
        request.ContentType = "application/x-www-form-urlencoded";
        System.IO.Stream reqStrm = request.GetRequestStream(); // create a new web request stream
        int Lg = requestContent.Length;                         // set the request's content length to the length of the byte array
        reqStrm.Write(requestContent, 0, Lg);
        System.Net.WebResponse webResponse = request.GetResponse();
        int rspLg = (int)webResponse.ContentLength;          // this value is set by "Content-Length" header recieved from the server
        System.IO.Stream rspStrm = webResponse.GetResponseStream();
        int tot = 0;
        if (rspLg > 0)
        {
            allBytes = new byte[rspLg];         // change the size of the response array to the size of the response contents
            int buffSize = 1000;                // set the buffer size to 1000 bytes
            do
            {
                byte[] buffer = new byte[buffSize];
                // the number of bytes read from the response stream is assigned to rdl
                // rdl might be smaller than bufferSize. In this case the rest of the buffer array contains 0 bytes (or null bytes)
                int rdl = rspStrm.Read(buffer, 0, buffSize);
                if (rdl > 0)
                {
                    // use Buffer.BlockCopy to copy a bloc of bytes with a length of rdl starting at index 0 from buffer
                    // into another array at index tot
                    // If you want to use data as they arrive you can use BlockCopy to copy data from buffer into another array
                    // The use a function such as ProcessData to operate on the data
                    byte[] block = new byte[rdl];       // the block has a legnth equal to rdl
                    System.Buffer.BlockCopy(buffer, 0, block, 0, rdl);
                    ProcessData(block);     
                    // tot is the total size of the data recieved from the server so far
                    // the following line copies bytes from buffer (at index 0 to rdl) to allBytes array at index tot
                    System.Buffer.BlockCopy(buffer, 0, allBytes, tot, rdl);
                    tot += rdl;
                }
            } while (tot < rspLg);
        }
        reqStrm.Close();
        rspStrm.Close();
        webResponse.Close();
        string rspStr = System.Text.UTF8Encoding.UTF8.GetString(allBytes);
        Console.WriteLine($"The following data was recieved from {url}");
        Console.WriteLine(rspStr);
    }

    return allBytes.ToList();
} // end fuction

public void ProcessData(byte[] data)
{
    // do something with data here
}

I also suggest using asynchronous programming by using Threads (System.Threading.Thread) along with async, and await keywords. So that, the main program thread is not blocked and your app remains responsive while data is being downloaded. You can have a look at multi threading", callback functions, and asynchronous programming.

Leeland