download file client-side chunk by chunk
Asked Answered
L

1

8

I'm using WebRTC to send a file to a connected peer, and I'm sending the file in chunks. However, I'm having trouble figuring out how to get the peer to save/download the file as it is streaming in, chunk by chunk.

All the examples I've found online recommend doing something like this:

// sender
dataConnection.send({
   'file': file
});

// receiver
dataConnection.on('data', function(fileData) {

    var dataView = new Uint8Array(fileData);
    var dataBlob = new Blob([dataView]);
    var url = window.URL.createObjectURL(dataBlob);

    // create <a>
    var link = document.createElement('a');
    link.href = url;
    link.download = fileName;
    document.body.appendChild(link);

    // trigger the download file dialog
    link.click();
}

This approach, however, doesn't support getting chunks of the file and writing each chunk as it comes... it must wait for the entire file to be read on the sender's side and sent to the receiver.

What I'm trying to do is something like this:

// sender
for (var i = 0; i < fileSize; i += chunkSize) {

    var fileReader = new FileReader();

    // read next chunk
    var blob = file.slice(start, end);
    ...
    fileReader.onload = function(e) {
        ...
        dataConnection.send({ 'blob': blob });
    }
    fileReader.readAsArrayBuffer(blob);
}

This way, I'm reading the file chunk by chunk and sending each chunk to the receiver as I read it. Now, the only way I know how to actually save a file that is being sent in this way is by doing what is described in this blog post:

http://bloggeek.me/send-file-webrtc-data-api

... described in "Step 6: Downloading to regular FS". However, this approach takes all the chunks as they come in, stores them in memory, then builds a large UInt8Array in memory and then lets the receiver download the file. This is really heavy on memory and is realistically limited to maybe a couple hundred MB, so it doesn't scale.

Is there a way to open the file download dialog after the first chunk comes in, and keep writing the chunks as they come in so that the download is a "streaming" download?

Lactase answered 16/5, 2014 at 18:21 Comment(8)
I don't think that is possible, at least not with plain javascript, as there is no way to write to the blob after it has been created. You can however store it in the localStorage filesystem, but only chrome supports it and specs are deprecated since a month because no other browser adapted it. This should prevent memory overflow, but you need to know the size of the total file that is being send before creating the storage.Glauconite
I'm using WebRTC only for Chrome, so that would be ok. Can you give an example of how localstorage would work, or link to some examples?Lactase
@Glauconite "as there is no way to write to the blob after it has been created" It is possible to write to a Blob, or, rather create a new Blob using bytes from existing Blob using Blob.prototype.slice()Desmond
@Desmond It is indeed possible to create a Blob which is a concatenation of other Blobs (new Blob([blobA, blobB, blobC])), you don't even need .slice() for that (unless you meant using that function on the sender side, of course). However, this doesn't answer the question; the asker is looking for a method to save the chunks to disk as they come in, instead of accumulating them somewhere (these Blob 'tricks', for example) before downloading the entire file through the webbrowser (using Blobs is probably better than using a UInt8Array, though).Glauconite
@Glauconite See github.com/jimmywarting/StreamSaver.jsDesmond
@Glauconite See also JavaScript: Writing to download stream , Is it possible to append bytes to a data URI at href of <a> element having download attribute after clicking element?Desmond
@Desmond I was just reading that code and about to post exactly the same link, seems this question finally has a valid answer after about two years.Glauconite
@Glauconite "seems this question finally has a valid answer after about two years." A good candidate for a canonical Question/Answer?Desmond
L
5

UPDATE

Streams API: https://streams.spec.whatwg.org

https://jakearchibald.com/2016/streams-ftw


Unfortunately, from what I've researched, there is no way to open the file save / file download dialog and save/download a file in a "streaming" fashion.

The approach I will take is to use the FileSystem API. Unfortunately, this isn't fully supported by every browser:

... and it doesn't seem likely that many browsers will adopt this API :(

Lactase answered 5/6, 2014 at 3:41 Comment(2)
See JavaScript: Writing to download streamDesmond
See #39959967Desmond

© 2022 - 2024 — McMap. All rights reserved.