Difference between Web Streams and Node.js Stream APIs

Question

I'm building a file upload application to familiarize myself with the concept of streams. I'm trying to turn a file list or blobs into a stream then upload it to the backend and store it on the file system.

I had no issue with frontend and backend implementations respectively but I'm having a hard time connecting the two. My main problem is that I don't understand the difference between the Web Streams API and the Node.js Streams API. I managed to turn the blobs of selected input files in the browser into a Web ReadableStream but the packages I tried (axios for requests, socket.io and socket.io-stream for WebSocket) only accept the Node.js version Stream as arguments. I also could not pipe a Web ReadableStream into a Node.js Writeable or Duplex Stream. The method names are also different (e.g.: pipeTo or pipeThrough in Web API and pipe in Node.js API).

I know there are implementation differences between Node.js and browsers but naively, I thought the APIs would be similar. Can I somehow trivially convert between Web streams and browserified Node.js streams and I'm missing something? Does it worth using the Web Stream API over stream-browserify?

Berl

This needs some sample code. I can't tell whether OP's confusion lies on the front-end or the back-end. You can't use Node streams in browserland, and for that reason I assume there is no browserland library that expects a Node stream. On the back-end, you're only dealing with Node streams: even data streamed to the server using a browserland stream would present as a Node stream on the server side. Without some code, it's hard to imagine where in all creation OP's problem could even exist. – Bathymetry

axios probably has just confusing types, their browser API allows only FormData, File, and Blob, in addition to the common ones like string, ArrayBuffer, etc. Their server API has Stream and Buffer though, so I thought I can use this on the frontend too. I can't remember why I couldn't make socket.io work, you're right that I should have added sample code. – Berl

Answer 1

It's not too difficult to convert a web stream to a Node.js stream manually, but you should really try to find a library that accepts native web streams instead of shoehorning a Node.js shim for the stream built-in into the browser with Browserify.

However, if it proves necessary to use a Node.js stream shim in the browser, you need to install stream-browserify and use like this:

import { Readable, Writable } from 'stream-browserify;'

// window.ReadableStream to Node.js Readable
const webRSToNodeRS = rs => {
  const reader = rs.getReader();
  const out = new Readable();
  reader.read().then(async ({ value, done }) => {
    while (!done) {
      out.push(value);
      ({ done, value } = await reader.read());
    }
    out.push(null);
  });
  return out;
}

// window.WritableStream to Node.js Writable
const webWSToNodeWS = ws => {
  const writer = ws.getWriter();
  const out = new Writable();
  out._write = (chunk, encoding, callback) => {
    writer.write(chunk);
    callback();
  };
  out._final = callback => {
    writer.close();
    callback();
  };
  return out;
}

These methods should be enough to have full interop between web and Node streams. For example, if you want to pipe a web ReadableStream to a Node.js Writable/Duplex:

const pipeWebRSToWritable = (rs, writable) => {
  // After converting you can use the normal pipe methods
  webRSToNodeRS(rs).pipe(writable);
}

However I'd like to mention that you don't need a library to stream data from the client to the server. The fetch API natively supports web streams and is probably the way you should go.

// POST a ReadableStream (perhaps of a file) to the server
// Way easier and more performant than using a 3rd party library...
const postRSToServer = rs => fetch('/your/server/endpoint', {
  method: 'POST',
  body: rs
});

Last note: make sure you're directly using the Blob.prototype.stream method (call this on a File object, e.g. file.stream(), since File extends Blob). There are some ways to get a ReadableStream from a file in JS that actually end up loading all of the file into memory in the browser, which you don't want.

Fossorial

Thanks for the detailed explanation! My use case is that I encrypt files in the browser then upload the encrypted chunks over WebSocket. Sending larger files (>5GB) in requests didn't work for me reliably, probably because of my server setup, so I switched to WebSocket but I'll definitely revisit using requests. – Berl

({ done, value }) = await reader.read(); gives me Invalid parenthesized assignment pattern in babel and SyntaxError: Invalid left-hand side in assignment in Node.js. Is that valid syntax? – Injun

You're right, you need parentheses around the entire assignment. I've updated the answer. – Fossorial

Answer 2

As per docs, since version 16, Node.js includes a (still experimental) implementation of the WebStreams API.

And the old Stream API has now (since v. 17) gained some toWeb/FromWeb methods to bridge both.

Some info here.

Chemo

Answer 3

For those (like myself) hitting this question in regard to the streams returned by node-fetch, note that undici is a web-compatible implementation of fetch that may eventually become the official node implementation.

Also, see leonbloy's answer RE the new toWeb/fromWeb methods, and WebStreams API.

Alga