Uploading HTTP progress tracking

Question

I've got WPF application I'm writing that posts files to one of social networks. Upload itself working just fine, but I'd like to provide some indication of how far along I am with the uploading.

I tried a bunch of ways to do this:

1) HttpWebRequest.GetStream method:

using (
 var FS = File.Open(
  localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Method = "POST";
    request.ProtocolVersion = HttpVersion.Version11;
    request.ContentType = "multipart/form-data; boundary=--AaB03x";
    //predata and postdata is two byte[] arrays, that contains
    //strings for MIME file upload (defined above and is not important)
    request.ContentLength = predata.Length + FS.Length + postdata.Length;
    request.AllowWriteStreamBuffering = false;
    using (var reqStream = request.GetRequestStream())
    {
        reqStream.Write(predata, 0, predata.Length);
        int bytesRead = 0;
        int totalRead = 0;
        do
        {
            bytesRead = FS.Read(fileData, 0, MaxContentSize);
            totalRead += bytesRead;
            reqStream.Write(fileData, 0, bytesRead);
            reqStream.Flush(); //trying with and without this
            //this part will show progress in percents
            sop.prct = (int) ((100*totalRead)/len);
        } while (bytesRead > 0);
        reqStream.Write(postdata, 0, postdata.Length);
    }
    HttpWebResponse responce = (HttpWebResponse) request.GetResponse();
    using (var respStream = responce.GetResponseStream())
    {
        //do things
    }
}

2) WebClient way (much shorter):

void UploadFile (url, localFilePath)
{
    ...
    WebClient client = new WebClient();
    client.UploadProgressChanged += new UploadProgressChangedEventHandler(UploadPartDone);
    client.UploadFileCompleted += new UploadFileCompletedEventHandler(UploadComplete);
    client.UploadFileAsync(new Uri(url), localFilePath);
    done.WaitOne();

    //do things with responce, received from UploadComplete
    JavaScriptSerializer jssSer = new JavaScriptSerializer();
    return jssSer.Deserialize<UniversalJSONAnswer>(utf8.GetString(UploadFileResponce));
    //so on...
    ...
}

void UploadComplete(object sender, UploadFileCompletedEventArgs e)
{
    UploadFileResponce=e.Result;
    done.Set();
}

void UploadPartDone(object sender, UploadProgressChangedEventArgs e)
{
    //this part expected to show progress
    sop.prct=(int)(100*e.BytesSent/e.TotalBytesToSend);
}

3) Even TcpClient way:

using (
 var FS = File.Open(
  localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    long totalRead = 0;
    using (var client = new TcpClient(urli.Host, urli.Port))
    {
        using (var clearstream = client.GetStream())
        {
            using (var writer = new StreamWriter(clearstream))
            using (var reader = new StreamReader(clearstream))
            {
                //set progress to 0
                sop.prct = 0;
                // Send request headers
                writer.WriteLine("POST " + urli.AbsoluteUri + " HTTP/1.1");
                writer.WriteLine("Content-Type: multipart/form-data; boundary=--AaB03x");
                writer.WriteLine("Host: " + urli.Host);
                writer.WriteLine("Content-Length: " + (predata.Length + len + postdata.Length).ToString());
                writer.WriteLine();
                //some data for MIME
                writer.Write(utf8.GetString(predata));
                writer.Flush();
                int bytesRead;
                do
                {
                    bytesRead = FS.Read(fileData, 0, MaxContentSize);
                    totalRead += bytesRead;
                    writer.BaseStream.Write(fileData, 0, bytesRead);
                    writer.BaseStream.Flush();
                    sop.prct = (int) ((100*totalRead)/len);
                } while (bytesRead > 0)
                writer.Write(utf8.GetString(postdata));
                writer.Flush();
                //read line of response and do other thigs...
                respStr = reader.ReadLine();
                ...
            }
        }
    }
}

In all cases the file was successfully sent to the server. But always progress looks like this: for a few seconds it runs from 0 to 100 and then waits until file actually uploading (about 5 minutes - file is 400MB).

So I think the data from a file is buffered somewhere and I'm tracking not uploading, but buffering data. And then must wait until it's uploaded.

My questions are:

1) Is there any way to track actual uploading data? That the method Stream.Write() or Flush() (which as I read somewhere, does not work for NetworkStream) did not return until it receives confirmation from the server that the TCP packets received.

2) Or can I deny buffering (AllowWriteStreamBUffering for HttpWebRequest doesn't work)?

3) And does it make sense to go further "down" and try with Sockets?

updated:

To avoid any doubts in the way of progress displaying on UI, I rewrote the code to log a file. so, here is code:

using (var LogStream=File.Open("C:\\123.txt",FileMode.Create,FileAccess.Write,FileShare.Read))
using (var LogWriter=new StreamWriter(LogStream))
using (var FS = File.Open(localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Timeout = 7200000; //2 hour timeout
    request.Method = "POST";
    request.ProtocolVersion = HttpVersion.Version11;
    request.ContentType = "multipart/form-data; boundary=--AaB03x";
    //predata and postdata is two byte[] arrays, that contains
    //strings for MIME file upload (defined above and is not important)
    request.ContentLength = predata.Length + FS.Length + postdata.Length;
    request.AllowWriteStreamBuffering = false;
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " Start write into request stream. ");
    using (var reqStream = request.GetRequestStream())
    {
        reqStream.Write(predata, 0, predata.Length);
        int bytesRead = 0;
        int totalRead = 0;
        do
        {
            bytesRead = FS.Read(fileData, 0, MaxContentSize);
            totalRead += bytesRead;
            reqStream.Write(fileData, 0, bytesRead);
            reqStream.Flush(); //trying with and without this
            //sop.prct = (int) ((100*totalRead)/len); //this part will show progress in percents
            LogWriter.WriteLine(DateTime.Now.ToString("o") + " totalRead= " + totalRead.ToString() + " / " + len.ToString());
        } while (bytesRead > 0);
        reqStream.Write(postdata, 0, postdata.Length);
    }
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " All sent!!! Waiting for responce... ");
    LogWriter.Flush();
    HttpWebResponse responce = (HttpWebResponse) request.GetResponse();
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " Responce received! ");
    using (var respStream = responce.GetResponseStream())
    {
        if (respStream == null) return null;
        using (var streamReader = new StreamReader(respStream))
        {
            string resp = streamReader.ReadToEnd();
            JavaScriptSerializer jssSer = new JavaScriptSerializer();
            return jssSer.Deserialize<UniversalJSONAnswer>(resp);
        }
    }
}

and here is result (I cut the middle):

2011-11-19T22:00:54.5964408+04:00 Start write into request stream. 
2011-11-19T22:00:54.6404433+04:00 totalRead= 1048576 / 410746880
2011-11-19T22:00:54.6424434+04:00 totalRead= 2097152 / 410746880
2011-11-19T22:00:54.6434435+04:00 totalRead= 3145728 / 410746880
2011-11-19T22:00:54.6454436+04:00 totalRead= 4194304 / 410746880
2011-11-19T22:00:54.6464437+04:00 totalRead= 5242880 / 410746880
2011-11-19T22:00:54.6494438+04:00 totalRead= 6291456 / 410746880
.......    
2011-11-19T22:00:55.3434835+04:00 totalRead= 408944640 / 410746880
2011-11-19T22:00:55.3434835+04:00 totalRead= 409993216 / 410746880
2011-11-19T22:00:55.3464837+04:00 totalRead= 410746880 / 410746880
2011-11-19T22:00:55.3464837+04:00 totalRead= 410746880 / 410746880
2011-11-19T22:00:55.3464837+04:00 All sent!!! Waiting for responce... 
2011-11-19T22:07:23.0616597+04:00 Responce received! 

as you can see program thinks that it uploaded ~400MB for about 2 seconds. And after 7 minutes file actually uploads and I receive responce.

updated again:

Seems to this is happening under WIndows 7 (not shure about x64 or x86). When I run my code uder XP everything works perfectly and progress is shown absolute correctly

Whiffet

Answer 1

it's more than year since this question was posted, but I think my post can be usefull for someone.

I had the same problem with showing progress and it behaved exactly like you described. So i decided to use HttpClient which shows upload progress correctly. Then I've encountered interesting bug - when I had Fiddler launched HttpClient started to show its upload progress in unexpected way like in WebClient/HttpWebRequest above so I thinked maybe that was a problem of why WebClient showed upload progres not correctly (I think I had it launched). So I tried with WebClient again (without fiddler-like apps launched) and all works as it should, upload progress has correct values. I have tested in on several PC with win7 and XP and in all cases progress was showing correctly.

So, I think that such program like Fiddler (probably not only a fiddler) has some affect on how WebClient and other .net classes shows upload progress.

this discussion approves it:

HttpWebRequest doesn't work except when fiddler is running

Holocene

ok. It's seems to be plausible. I really could do tests with fiddler opened. I'll check again when I get home, thx! – Whiffet

i think fiddler sets some system hooks that makes all .net network classes work like they shouldn't. – Holocene

Well, Fiddler enlists as a a HTTP proxy as far as I know, so it will have it's own buffers and acknowledge packets to the uploading host even before they're sent to the destination server. The inverse happens when downloading files while Fiddler is enabled (progress stays at 0% for a long time, then jumps to 100% quickly as the downloaded file is streamed from the proxy to the host that initiated the request). – Workmanship

Answer 2

You could use the WebClient's UploadFile to upload file rather than using writing file as a file stream. In order to track the percentage of the data received and uploaded you can use UploadFileAsyn and subscribe to its events.

In the code bellow I've used UploadFileAsyn to the upload files synchronously, but it need not to be synchronous as far as you don't dispose the instance of the uploader.

class FileUploader : IDisposable
{
    private readonly WebClient _client;
    private readonly Uri _address;
    private readonly string _filePath;
    private bool _uploadCompleted;
    private bool _uploadStarted;
    private bool _status;

    public FileUploader(string address, string filePath)
    {
        _client = new WebClient();
        _address = new Uri(address);
        _filePath = filePath;
        _client.UploadProgressChanged += FileUploadProgressChanged;
        _client.UploadFileCompleted += FileUploadFileCompleted;
    }

    private void FileUploadFileCompleted(object sender, UploadFileCompletedEventArgs e)
    {
        _status = (e.Cancelled || e.Error == null) ? false : true;
        _uploadCompleted = true;
    }

    private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
    {
        if(e.ProgressPercentage % 10 == 0)
        {
            //This writes the pecentage data uploaded and downloaded
            Console.WriteLine("Send: {0}, Received: {1}", e.BytesSent, e.BytesReceived);
            //You can have a delegate or a call back to update your UI about the percentage uploaded
            //If you don't have the condition (i.e e.ProgressPercentage % 10 == 0 )for the pecentage of the process 
            //the callback will slow you upload process down
        }
    }

    public bool Upload()
    {

        if (!_uploadStarted)
        {
            _uploadStarted = true;
            _client.UploadFileAsync(_address, _filePath);
        }
        while (!_uploadCompleted)
        {
            Thread.Sleep(1000);
        }
        return _status;
    }

    public void Dispose()
    {
        _client.Dispose();
    }
}

Client Code:

            using (FileUploader uploader = new FileUploader("http://www.google.com", @"C:\test.txt"))
        {
            uploader.Upload();
        }

You can register a custom callback (may be a delegate) on the FileUploadProgressChanged event handler to update your WPF UI.

The upload progress changed event get called more often if your callback for the event does any IO then that'll slowdown the download progress. It's best to have infrequent update e.g. the following code update only evey 10% up.

    private int _percentageDownloaded;

    private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
    {
        if (e.ProgressPercentage % 10 == 0 && e.ProgressPercentage > _percentageDownloaded)
        {

            _percentageDownloaded = e.ProgressPercentage;
            //Any callback instead of printline
            Console.WriteLine("Send: {0} Received: {1}", e.BytesSent, e.BytesReceived);
        }
    }

Phobia

thanks for your attention, I'll try your code tomorrow. But it seems to me that will not be any difference compared to my method 2) described above. – Whiffet

Yes it's, see the changes I've done recently. The bottom of the code. Limiting the callback on progress change will increase performance (when your callback has some IO related stuff). – Phobia

The problem still exists: when I tried to upload 400MB file, the progress bar changes from 0 to 50% in 10 seconds (it can't be uploaded so fast), then it freezes for 5 minutes and then (when file is actually uploaded) progress goes from 50 to 100 less than in one second. – Whiffet

Then the problem must be at the receiving end (the server/service you are uploading the file to) – Phobia

Answer 3

my suggestion is to use new HTTPClient class (available in .NET 4.5). It supports progress.

This article helped me a lot with this: http://www.strathweb.com/2012/06/drag-and-drop-files-to-wpf-application-and-asynchronously-upload-to-asp-net-web-api/

My code for upload file:

    private void HttpSendProgress(object sender, HttpProgressEventArgs e)
    {
        HttpRequestMessage request = sender as HttpRequestMessage;
        Console.WriteLine(e.BytesTransferred);
    }

    private void Window_Loaded_1(object sender, RoutedEventArgs e)
    {
        ProgressMessageHandler progress = new ProgressMessageHandler();
        progress.HttpSendProgress += new EventHandler<HttpProgressEventArgs>(HttpSendProgress);

        HttpRequestMessage message = new HttpRequestMessage();
        StreamContent streamContent = new StreamContent(new FileStream("e:\\somefile.zip", FileMode.Open));

        message.Method = HttpMethod.Put;
        message.Content = streamContent;
        message.RequestUri = new Uri("{Here your link}");

        var client = HttpClientFactory.Create(progress);

        client.SendAsync(message).ContinueWith(task =>
        {
            if (task.Result.IsSuccessStatusCode)
            { 

            }
        });
    }

Holocene

Answer 4

This one has been bugging me for at least one day. I have started with using WebClient.UploadFileAsync, next tried the ProgressMessageHandler for HttpClient then rolled my own HttpContent for the HttpClient API. None of those approaches worked (for me).

It appears HttpWebRequest, which sits at the bottom of most (all?) .NET Http abstraction like WebClient and HttpClient, buffers the request and response stream by default, which I confirmed by looking at it in ILSpy.

As others have noted, you can make your request use chunked encoding one way or another which will effectively disable buffering the request stream, but still this is not going to fix the progress reporting.

I found that it was necessary to flush the request stream after each block that I send in order to accurately reflect sending progress, or else your data will simply be buffered one step further down the pipeline (probably somewhere in NetworkStream or OS, didn't check). The sample code below works for me and also does a minimalistic job at translating back from a HttpWebResponse to HttpResponseMessage (which you may not need, YMMV).

public async Task<HttpResponseMessage> UploadFileAsync( string uploadUrl, string absoluteFilePath, Action<int> progressPercentCallback )
    {
        var length = new FileInfo( absoluteFilePath ).Length;

        var request = new HttpWebRequest( new Uri(uploadUrl) ) {
            Method = "PUT",
            AllowWriteStreamBuffering = false,
            AllowReadStreamBuffering = false,
            ContentLength = length
        };

        const int chunkSize = 4096;
        var buffer = new byte[chunkSize];

        using (var req = await request.GetRequestStreamAsync())
        using (var readStream = File.OpenRead(absoluteFilePath))
        {
            progressPercentCallback(0);
            int read = 0;
            for (int i = 0; i < length; i += read)
            {
                read = await readStream.ReadAsync( buffer, 0, chunkSize );
                await req.WriteAsync( buffer, 0, read );
                await req.FlushAsync(); // flushing is required or else we jump to 100% very fast
                progressPercentCallback((int)(100.0 * i / length));
            }
            progressPercentCallback(100);
        }

        var response = (HttpWebResponse)await request.GetResponseAsync();
        var result = new HttpResponseMessage( response.StatusCode );
        result.Content = new StreamContent( response.GetResponseStream() );

        return result; 
    }

Workmanship

Answer 5

At fast guess, you are running this code on UI thread. You need to run upload stuff on new thread. At that point you have 2 options. 1) You run timer on UI thread and update UI. 2) You update UI using Invoke(because you can't access UI from another thread) calls to update UI.

Sankey

no, all this stuff is running on its own thread. It's WPF binding and UI updates as I change sop.prct property – Whiffet

I watched progress at different ways, include step-by-step in debug mode. For instance, with TcpClient 99% of time gets this line: 'respStr = reader.ReadLine();' while everything else is runs in less than a minute – Whiffet

I dont see(i'm blind or that is defined in other place) how big is fileData byte[] array. In any way it should not be bugger than 16 - 256kb. – Sankey

Also you probably may not access UI from worker theread. For more info see: msdn.microsoft.com/en-us/magazine/cc163328.aspx – Sankey

yes, fileData defined in onther place and its lenght was 1MB. I just tried smaller sizes, but it did not help. And everything is fine with displaying progress in UI. I can actually see how progress bar goes from 0% to 100%. My problem is that most of the code above is executed for a few seconds, regardless of file size, and then hangs until the end of uploading on waiting for a response. And I can't actually understand when the server received part of data – Whiffet

You missed point, you are accessing currently UI from worker thread, thats not allowed. I provided link, what shows how todo it right. It all realted to it, for testing, you can disable UI updating code and write totalRead + SP + DateTime.Now CRLF to file. You can see that read/write loops happen all the time and won't hang. – Sankey

Updated question to allay all doubts about UI – Whiffet

Have you tried to set request.SendChunked = true. This will force Write method to send data when you pass it. – Sankey

Answer 6

In the first example I think your progress bar is showing how fast you write into the stream from the file on disk - not the actual upload progress (which is why it all happens to 100% really quickly then the upload chugs on*).

I might be wrong ^^ and have no WPF experience but I have uploaded massive files from Silverlight to WCF and the model used there is (as you do) to break up the file into blocks. Send each block. When you get a response from the server ("block 26 received ok"), update the progress bar as really, you can't (or should not) update the progress bar unless you /know/ that block x made it - and a good way to know that is if the server says it got it.

*I wish I could upload 400Mb in 5 mins. Would take me all day...

Lichfield

do not pay any attention to WPF, it is responsible only for the interface. I also think that it shows the progress of the writing to a buffer, which was then data sent to the server at the network level. So I need some event that fires only when part of data is actually sent to server or method that doesn't return until data is sent. Unfortunately, the server accepts only a single large http request, and I can't break up file at http level – Whiffet

Answer 7

I had the same problem. I spent a lot of time and solved the problem as follows: Antivirus AVAST. When I turn it off my program works perfectly...

Uppercut