HttpClient reading entire file before upload. UWP

Question

I'm making an UWP app that uploads files to facebook, I'm using a custom HttpContent to upload the files in 4k blocks to minimize the memory usage for big files (>100mb) and to report progress.

My custom HttpContent UploadWithProgressHttpContent:

 class UploadWithProgressHttpContent : HttpContent
{
    private readonly IProgress<OperationProgress> _progress;
    private readonly OperationProgress _data;
    private readonly Stream _file;
    private readonly int _bufferSize;
    private readonly CancellationToken _token;

    public UploadWithProgressHttpContent(
        IProgress<OperationProgress> progress,
    OperationProgress data,
    Stream file,
    int bufferSize,
    CancellationToken token)
{
    _progress = progress;
    _data = data;
    _file = file;
    _bufferSize = bufferSize;
    _token = token;
}



protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
    return CopyStreamWithProgress(_file, stream, _progress, _token, _data, _bufferSize);
}

public static async Task<Stream> CopyStreamWithProgress(
    Stream source,
    Stream destination,
    IProgress<OperationProgress> progress,
    CancellationToken token,
    OperationProgress progressData,
    int bufferSize
    )
{
    int read, offset = 0;
    var buffer = new byte[bufferSize];
    using (source)
    {
        do
        {
            read = await source.ReadAsync(buffer, 0, bufferSize, token);

            await destination.WriteAsync(buffer, 0, read, token);

            offset += read;
            progressData.CurrentSize = offset;
            progress.Report(progressData);
        } while (read != 0);
    }

    return destination;
}
}

What I'm experiencing (using fiddler) is that the whole file gets putted in memory before the upload starts (my progress meter reaches 100% before the upload even starts).

I did try setting the TransferEncodingChunked to true, and setting the file content length but the issue remains.

The upload source is inside a PCL (if it matters). I'm using the latest version of System.Net.Http. If need I'm using this the exact same way as it is used in the MediaFire SDK

Thanks for any help.

EDIT: Added the HttpClient usage:

public async Task<T> Upload<T>(Stream fileStream, string fileName)
{
    var handler = new HttpClientHandler();


    var cli = new HttpClient(handler);

    foreach (var header in Headers)
    {
        cli.DefaultRequestHeaders.Add(header.Key, header.Value);
    }

    var parameters = new MultipartFormDataContent();
    foreach (var parameter in Parameters)
    {
        parameters.Add(new StringContent(parameter.Value), parameter.Key);
    }

    if (fileStream != null)
    {
        var fileContent = new UploadWithProgressHttpContent(ProgressOperation, ProgressData, fileStream,
            _chunkBufferSize, Token, fileStream.Length);

        fileContent.Headers.ContentType = new MediaTypeHeaderValue(MimeTypeHelper.GetMimeType(fileName));
        fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue(StreamParamName);
        fileContent.Headers.ContentDisposition.FileName = fileName;
        fileContent.Headers.ContentLength = fileStream.Length;
        parameters.Add(fileContent, StreamParamName);
    }

    var req = new HttpRequestMessage(method, Path) { Content = parameters };
    if (fileStream != null)
        req.Headers.TransferEncodingChunked = true;


    var completionOption = HttpCompletionOption.ResponseContentRead;


    var resp = await cli.SendAsync(req, completionOption, Token).ConfigureAwait(false);

    return await DeserializeObject<T>(resp);
}

Gary

I couldn't fix this issue with a custom HttpContent, my alternative was using a StreamContent, and making a Stream Decorator that reported progressed on read. By using a StreamContent HttpClient streams the content without bringing it (all) to memory. – Gary

Not exactly, I'm using a custom httpcontent, I already check that using a stream content it works as expected, I just want to know why it doesn't work with a custom httpcontent, or if there is any way it would. – Gary

Can you share also the piece code around HttpClient ? Btw. I would consider using wire shark instead (not sure if adding fiddler as http proxy isn't messing with the traffic). – Longmire

Done, thanks for the help! – Gary

Did you try with an override for HttpContent.TryComputeLength? – Haystack

Side note, might want to change your HttpClient to a single instance to be used by this call. See here – Bullis

I'm aware that HttpClient should have instances pooled, but since I'm on a multi-thread environment I didn't want to deal with synchronization while I build the app. @Uwe I did try that without success. – Gary

Answer 1

You have the same problem as quantum mechanics - the act of observing changes the observed. Fiddler does not support request streaming - see Fiddler makes HttpWebRequest/HttpClient behaviour unexpected and http://www.telerik.com/forums/is-it-possible-to-not-buffer-requests

Using wireshark I can see the chunks.

Longmire

Not quite, If I disable fiddler the issue remains. – Gary

Answer 2

After hours wasted on this issue, it seams that this a HttpClient implementation issue. So if you want stream content to a server (and report progress) the best alternative is to use a StreamContent and decorate the reads to report the progress.

NOTE: This is true to the 4.3.1 version of System.Net.Http and version 2.2.29 Microsoft.Net.Http nugget packages

Gary