How to pipe one readable stream into two writable streams at once in Node.js?

Question

The goal is to:

1. Create a file read stream.
2. Pipe it to gzip (zlib.createGzip())

3. Then pipe the read stream of zlib output to:

   1) HTTP response object

   2) and writable file stream to save the gzipped output.

Now I can do down to 3.1:

var gzip = zlib.createGzip(),
    sourceFileStream = fs.createReadStream(sourceFilePath),
    targetFileStream = fs.createWriteStream(targetFilePath);

response.setHeader('Content-Encoding', 'gzip');

sourceFileStream.pipe(gzip).pipe(response);

... which works fine, but I need to also save the gzipped data to a file so that I don't need to regzip every time and be able to directly stream the gzipped data as a response.

So how do I pipe one readable stream into two writable streams at once in Node?

Would sourceFileStream.pipe(gzip).pipe(response).pipe(targetFileStream); work in Node 0.8.x?

Overhear

Answer 1

I found that zlib returns a readable stream which can be later piped into multiple other streams. So I did the following to solve the above problem:

var sourceFileStream = fs.createReadStream(sourceFile);
// Even though we could chain like
// sourceFileStream.pipe(zlib.createGzip()).pipe(response);
// we need a stream with a gzipped data to pipe to two
// other streams.
var gzip = sourceFileStream.pipe(zlib.createGzip());

// This will pipe the gzipped data to response object
// and automatically close the response object.
gzip.pipe(response);

// Then I can pipe the gzipped data to a file.
gzip.pipe(fs.createWriteStream(targetFilePath));

Overhear

Downvote. This is redundant, doesn't add any new information at all and actually manages to increase confusion. – Cowage

You cannot pipe writeable streams to anything: github.com/nodejs/readable-stream/blob/master/lib/… you need a douplex or a readable stream to do that. – Greenhouse

Answer 2

Pipe chaining/splitting doesn't work like you're trying to do here, sending the first to two different subsequent steps:

sourceFileStream.pipe(gzip).pipe(response);

However, you can pipe the same readable stream into two writeable streams, eg:

var fs = require('fs');

var source = fs.createReadStream('source.txt');
var dest1 = fs.createWriteStream('dest1.txt');
var dest2 = fs.createWriteStream('dest2.txt');

source.pipe(dest1);
source.pipe(dest2);

Allusion

Pipes are chainable. Checkout the Zlib doc nodejs.org/api/zlib.html. You can see raw.pipe(zlib.createGzip()).pipe(response); I know the solution you've given, but it doesn't solve my problem because in my case I don't have a specific read stream. The data is generated by zlib at runtime, and I need to pipe its data to two writable streams. – Overhear

Pipes are chainable? No if you take into account that the last pipe() doesn't work over the first raw stream. It's not like in jQuery that you chain to work on the same object. The last pipe(response) just takes the input from gzip not from raw. – Facula

As of 2018, I'm able to chain pipe() calls and get the data I expect. This is especially useful in something like an AWS Lambda function - read from a bucket as a stream; pipe to zlib gunzip; pipe to writable stream, which saves it to another bucket / key. – Traprock

This answer me helped understand that the stream ends when piping to fs.createWriteStream, so after that point it can't be chained any further. – Damascene

Answer 3

you can use "readable-stream-clone" package

const fs = require("fs");
const ReadableStreamClone = require("readable-stream-clone");

const readStream = fs.createReadStream('text.txt');

const readStream1 = new ReadableStreamClone(readStream);
const readStream2 = new ReadableStreamClone(readStream);

const writeStream1 = fs.createWriteStream('sample1.txt');
const writeStream2 = fs.createWriteStream('sample2.txt');

readStream1.pipe(writeStream1)
readStream2.pipe(writeStream2)

Pulpy