Download large data stream (> 1Gb) using javascript

Question

I was wondering if it was possible to stream data from javascript to the browser's downloads manager.

Using webrtc, I stream data (from files > 1Gb) from a browser to the other. On the receiver side, I store into memory all this data (as arraybuffer ... so the data is essentially still chunks), and I would like the user to be able to download it.

Problem : Blob objects have a maximum size of about 600 Mb (depending on the browser) so I can't re-create the file from the chunks. Is there a way to stream these chunks so that the browser downloads them directly ?

Bony

bugs.chromium.org/p/chromium/issues/detail?id=375297 seems to be the most appropriate example of what i'm currently facing (this example uses multiple files, but the same error occurs with 1 big file) ... It's from one year ago, so I would be glad to be proven wrong, but I think it hasn't been fixed yet ? – Bony

Was able to create and read var b = new Blob([new Uint8Array(500*1024*1024)], {type: 'application/octet-string'}); though var b = new Blob([new Uint8Array(500*1024*1024), new Uint8Array(500*1024*1024)], {type: 'application/octet-string'}); logged Uncaught RangeError: Array buffer allocation failed – Halftone

I have the same problem; I was planning to break up the stream into smaller files and offer each as a separate download. – Ormiston

Note, Blob error was not thrown, but Array buffer allocation failed. Asked this Question for a different purpose Where is Blob binary data stored?; eventually found the files at user filesystem where, presumably, Blob data is stored. Another option could be to create a .zip folder; or launch chromium with --unlimted-storage flag would change result Though have not tried with 1GB+ as of yet. Will try code at previous comment at firefox. May require user action at settings or preferences. – Halftone

I'm running Chromium Version 56.0.2924.76 (16 GB RAM & 20 GB free space on my hard drive). var b = new Blob([new Uint8Array(500*1024*1024), new Uint8Array(500*1024*1024)], {type: 'application/octet-string'}); didn't log any error for me, but trying to download the file using URL.createObjectURL(b) logged "Failed - No file" in the downloads manager ... – Bony

The limit should be 2GB on Chrome 57 which is 9 days from now. – Maxillary

Related Download large file >1GB using http protocol, java and javascript – Halftone

Brian Have not tried, though one approach could be to stream bytes to a document, instead of a Blob Method for streaming data from browser to server via HTTP, then use one of approaches posted by @LeoFarmer at How to download a file without using <a> element with download attribute or a server?. What is MIME type of file offered for download? – Halftone

Interestingly chrome://blob-internals listed 50 entries for single Blob created by var b = new Blob([new Uint8Array(500*1024*1024), new Uint8Array(500*1024*1024)], {type: 'application/octet-string'});. Is 50 the current limit for number of Length: 10,485,760 chunks allocated to one Blob? – Halftone

@Bony What is type of file that should be downloaded? – Halftone

Any type unfortunately ... I'll try your solution using a document tomorrow and let you know how it goes. – Bony

@Bony See StreamSaver.js – Halftone

Is this for production facing the web ? or only for your own use case ? You can achieve it with the FileSystem API which is not anymore on specs tracks, but which is still supported by at least chrome. – Drabble

Yes, which is why the FileSystem API and StreamSaver.js could only be temporary fixes ... – Bony

In this case, for a more bullet proof solution, I would personally go to a save on server solution. From there you'll be able to concatenate your chunks on the fly, and once user wants to download it, he will do so from the server. – Drabble

Answer 1

Following @guest271314's advice, I added StreamSaver.js to my project, and I successfully received files bigger than 1GB on Chrome. According to the documentation, it should work for files up to 15GB but my browser crashed before that (maximum file size was about 4GB for me).

Note I: to avoid the Blob max size limitation, I also tried to manually append data to the href field of a <a></a> but it failed with files of about 600MB ...

Note II: as amazing as it might seem, the basic technique using createObjectURL works perfectly fine on Firefox for files up to 4GB !!

Bony

Can you include working example of implementation at Answer? – Halftone

Answer 2

if you want to fetch a large file blob from an api or url, you can use streamsaver.

npm install streamsaver

then you can do something like this

import { createWriteStream } from 'streamsaver';

export const downloadFile = (url, fileName) => {
  return fetch(url).then(res => {
    const fileStream = createWriteStream(fileName);
    const writer = fileStream.getWriter();
    if (res.body.pipeTo) {
      writer.releaseLock();
      return res.body.pipeTo(fileStream);
    }

    const reader = res.body.getReader();
    const pump = () =>
      reader
        .read()
        .then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));

    return pump();
  });
};

and you can use it like this:

const url = "http://urltobigfile";
const fileName = "bigfile.zip";

downloadFile(url, fileName).then(() => { alert('done'); });

Perot

Take into account that in order to accomplish streamSaver installs the service worker in a secure context hosted on github static pages source – Semipermeable

@AbrahamSimpson I came to say the same. This is a huge security risk. – Verdha