Save client generated data as file in JavaScript in chunks

Question

I'm developing a FileShare application with webRTC. I want to implement the client in JavaScript/HTML. The code should be run on the clients browser. I need to save them when downloaded via webRTC. The files can be quite big and I can't completely donwload them and save them in a array or blob before saving them to disk as a file.

Is there any API that allows me to save the file in chunks as I recieve them?

I have found so far Downloadify, FileSave.js and html5 FileWriterApi so far. While the first two are not chunked and require me to first download the complete file to memory before saving, the FileWriterAPI is not available on most browsers.

Abixah

help.infragistics.com/Help/NetAdvantage/ASPNET/2013.1/CLR4.0/… aurigma.com/docs/iu7/saving-uploaded-files-in-aspnet.htm I have this two links which may help you – Eberhard

For reasonably-sized files, my first instinct is to store each chunk in its own Blob, join them all together into one big Blob at the end (using the new constructor) and use createObjectURL to save the final Blob. IIRC, JS Blobs are stored to disk where necessary, so unless the files are really huge (or I'm really wrong ;) ), it shouldn't be too horrific? – Mausoleum

@JordanGray Might be a good idea - I assumed the downloaded chunks need to be persisted for continuous download between browser sessions, but if not, this might be the cleanest idea (if the browsers do implement persisting the blob to file when needed). – Baptist

Answer 1

As @jordan-gray suggested, saving the chunks in blobs and joining them to a larger blob might be a solution if:

• Persistence of chunks is not needed (i.e. closing the browser will delete all chunks)
• The file is persisted only by the user saving it to his own filesystem. The web application will not have access to the file once it is closed, unless the user gave access to the saved file again.
• Possibly, if the file sizes are not too big (you'll have to benchmark to find that out). Chrome was behaving quite nice for me for chunks totaling at 1GB.

I've created a simple test for using blobs as chunks. You can play around with the different size and chunk numbers parameters:

var chunkSize = 500000;
var totalChunks = 200;
var currentChunk = 0;
var mime = 'application/octet-binary';
var waitBetweenChunks = 50;

var finalBlob = null;
var chunkBlobs =[];

function addChunk() {
    var typedArray = new Int8Array(chunkSize);
    chunkBlobs[currentChunk] = new Blob([typedArray], {type: mime});
    console.log('added chunk', currentChunk);
    currentChunk++;
    if (currentChunk == totalChunks) {
        console.log('all chunks completed');
        finalBlob = new Blob(chunkBlobs, {type: mime});
        document.getElementById('completedFileLink').href = URL.createObjectURL(finalBlob);
    } else {
        window.setTimeout(addChunk, waitBetweenChunks);
    }
}
addChunk();

If you do need that persistence, the W3C File System API should support what you need. You can use it to write the chunks to separate files, and then when all the chunks are completed you can read them all and append them to a single file, and remove the chunks.

Note that it works by assigning a sandboxed filesystem for your application (for a given quota), and the files are only accessible to that application. If the files are meant to use outside of the web application, you might need the function for the use to save the file from the application filesystem to his "normal" filesystem. You can do something like that using the createObjectURL() method.

You are right about current state of browser support. A Filesystem API polyfill is available, which is based on IndexedDB (which is more widely supported) as a filesystem emulation backend. I did not test the polyfill on large files. You might run into size limits or performance limitations.

Baptist

Yup, this is pretty much what I had in mind. Based on my understanding of the spec it should be able to scale reasonably well, with chunks being persisted to underlying storage where necessary. :) – Mausoleum

It looks like this is still being saved to memory first (currentChunk == totalChunks) before saving to disk, correct? – Tarryn

No, think of new Blob() as if it were returning a pointer to an object on disk. developer.mozilla.org/en/docs/Web/API/Blob – Coben

@AmitayDobo How could it show the download process from the first chunk, I would like it to show the Browsers download behavior. Could it be done? – Goldwin

Answer 2

Did you check https://github.com/Peer5/Sharefest out ? It should cover your requirements

Myrlmyrle

seems to be an issue for them as well... github.com/Peer5/ShareFest/issues/53 – Abixah

Answer 3

while (currentChunk < totalChunks) {
                var offset = currentChunk * chunkSize;
        var currentFilePart = rr.files.item(0).slice(offset, (offset+chunkSize));
        var cFilePart = new FileReader();
        var ar1 = new Uint8Array(currentFilePart.arrayBuffer());
        for (var i = 0; i < chunkSize; i++) 
        {
                st = ar1[i] ^ bb;
                st = ar1[i];
                Rajar2[Rajtop] = iii;
                Rajtop++;
        }
                currentChunk++;
        iii = iii  + currentChunk;
        }
    {
        {
            var blob1 = new Blob([Rajar2], {type: "application/octet-stream"});
            var fileName = 'a.txt';
            var url = window.URL.createObjectURL(blob1);
            {
                var element = document.createElement('a');
                element.style.display = 'none';
                    element.href = url;
                    element.download = fileName;
                document.body.appendChild(element);
                element.click();
            }
        }
    }

Celindaceline

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Burack