How can I make a Readable stream from a Writable stream? (Node.js)

Question

I'm creating Excel data on the fly using Exceljs

I want to buffer this thing in memory, then send it off to the user using Koa.

The write method for Exceljs expects a writableStream:

workbook.xlsx.write(writableStream, options)

BUT Koa expects a readable stream:

response.body = readableStream

I know I can pipe a readable stream into a writable stream, but how do I do the opposite? I want Exceljs to write into the writable stream and have Koa read from the same stream. I'm so frustrated with the streams API!

Among 20 other things, I tried this:

const ReadableStream = require("memory-streams").ReadableStream

const reader = new ReadableStream()

const writer = new stream.Writable({
    write: function(chunk, encoding, next) {
        console.log(chunk.toString())
        // reader.push(chunk, encoding)
        next()
    }
})

const reader = new MemoryStream(null, {readable: true})
// reader.write = reader.unshift
const writer = reader

workbook.xlsx.write(writer, {})
return reader

But it doesn't work, I get some weird error about not being able to write to a stream that is closed. Even if I handle the error, however, my Excel file doesn't open.

So how can I make a readable stream out of a writable stream?

Lymphangitis

Answer 1

You need a transform stream. It's both readable and writable.

stream = new require('stream').Transform()
stream._transform = function (chunk,encoding,done) 
{
    this.push(chunk)
    done()
}

Exurb

I looked at the documentation and my head went numb. Any suggestions? I'm looking for something stupid like the equivalent of the identity function: a transform that doesn't do anything. – Lymphangitis

There are several things missing to understand what is going on... How do you use this Transform? How do you pipe it to a Writable/Readable streams (which was the original question)? – Loella

Answer 2

Have you considered Passthrough stream? That's a simple transform stream that passes all data passed to it, so you can read from a writable stream.

Natividadnativism

Great option if you just want to get an stream, modify the stream, and stream it to some other place – Debtor

Answer 3

For all who need to have a stream writable and readable. This is what finally worked for me.

Use case was QRCode package generating QRCode png image from dynamic var (ex: userId) only using stream.Writable --> to uploading to AWS S3 bucket only using stream.Readable.

import QRCode from 'qrcode';
import AWS from 'aws-sdk';
import { Transform } from 'stream';

const userId = 'userIdString';

// AWS Config, import this from a safe place!!!
AWS.config.update({
  accessKeyId: 'xxxxxxxxxx',
  secretAccessKey: 'xxxxxxxxx',
  region: 'xxxxxxx',
});

const s3 = new AWS.S3({ apiVersion: '2006-03-01' });

// Create Stream, Writable AND Readable
const inoutStream = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk);
    callback();
  },
});

// Need writable stream
QRCode.toFileStream(inoutStream, userId);

//Just to check
inoutStream.on('finish', () => {
  console.log('finished writing');
});

// Need readable stream
  s3.upload(
    {
      Bucket: 'myBucket',
      Key: `QR_${userId}.png`,
      Body: inoutStream,
      // ACL: 'public-read',
      ContentType: 'image/png',
    },
    (err, data) => {
      console.log(err, data);
    },
  )
  .on('httpUploadProgress', (evt) => {
    console.log(evt);
  })
  .send((err, data) => {
    console.log(err, data);
  });

If you never used stream before this is a very good source.

Actaeon

Typescript complains that a Transform is not the same as a WriteableStream. I can get arround with using as any. But is it safe to assume all methods of WritableStream are available in the Transform as well? – Reed