Pass large amounts of data between web worker and main thread

Question

Is there a way to pass large amounts of data (multiple MB) between a web worker and the main thread? I work in a project where I need to download files, modify them a bit and then somehow let the user download the modified file. I found the following ways to pass data between a web worker and the main UI

1. Using the regular postMessage method.
2. Using transferable objects (Chrome only)
3. Create a URL reference to a blob and only send the URL (works in Chrome, in others as well?)

I think (1) is fine when sending smaller objects but it takes a lot of time and space when dealing with files larger than a few MB since it is serialized and sent as JSON. Chrome offers a way to transfer data using transferable objects where the data does not have to be copied. Unfortunately this is a Chrome only feature so far since it would have served my purpose otherwise.

The last thing I found was to, from the worker, create a URL to a blob using self.webkitURL and then only pass the URL reference to the UI. This works well, I can give the URL to the user and they can download the file. I have unfortunately not found a way to do this is in Firefox, is it possible?

Is there any other methods one could use to transfer data between workers and the main thread?

Norway

Answer 1

Firefox/Opera/Chrome all currently support a flavor of web workers called Transferable Objects which is extremely fast - also extremely easy to setup. Here we send to the ww (web worker) a browser allocated array which is populated by the ww and returned back to the browser side. This is passed by reference, not a copy : browser <-> ww

On the browser side :

var huge_array = new Float32Array(BIG_HUSKY_SIZE);

// worker.postMessage(huge_array.buffer);                      // old way
   worker.postMessage(huge_array.buffer, [huge_array.buffer]); // new Trans Obj

and then over inside the web worker :

self.onmessage = function(e) {

      var flt_arr = new Float32Array(e.data);

    //  typically you might want to populate flt_arr here

    //  now send data structure back to browser

    // self.postMessage(flt_arr.buffer);                    // old way
       self.postMessage(flt_arr.buffer, [flt_arr.buffer]); // new Trans Obj way
}

By simply putting the data object inside square brackets [in here] this tips off js to use Transferable Object mode. This also works when sending back and forth javascript objects containing multiple variables, not just typed arrays.

To quote :

Transferable objects are objects that are not copied (e.g. using something like structured cloning). Instead, the data is transferred from one context to another. The 'version' from the calling context is no longer available once transferred to the new context. For example, when transferring an ArrayBuffer from main app to Worker, the original ArrayBuffer from the main thread is cleared and no longer usable. This vastly improves performance of sending data to a Worker

http://html5-demos.appspot.com/static/workers/transferables/index.html https://html.spec.whatwg.org/multipage/workers.html

https://developer.chrome.com/blog/transferable-objects-lightning-fast/

As an example ... I use a Web Worker in this project https://github.com/scottstensland/websockets-streaming-audio to handle WebSocket streaming audio transport from server side to the browser ... as you can see js is not my lingua franca and that project was just a Proof of Concept for a much bigger project which watches video to synthesize the same information as audio ... and visa versa ... audio in -> a series of images out

Recurrent

Where did you get the quote? – Pieplant

the quote is from Eric Bidelman (Google staffer), it appears if you click on "Yo dawg, what is this?" which appears at top of URL I list above ... its parent link is updates.html5rocks.com/2011/12/… – Recurrent

Answer 2

According to this WebWorkers tutorial, WebWorkers now support to pass File and Blob objects and basically any object that can be used with the structured clone algorith... or at least Chrome does it, probably because it implments the FileSystem API. I don't know if it's the main reason but I hope don't and in fact this feature is implemented in other browsers... being able to process user selected files in background is a nice thing.

Irbm

Unfortunately that's still a copy operation even if it certainly increases the speed. – Norway