How to read an entire text stream in node.js?

Question

In RingoJS there's a function called read which allows you to read an entire stream until the end is reached. This is useful when you're making a command line application. For example you may write a tac program as follows:

#!/usr/bin/env ringo

var string = system.stdin.read(); // read the entire input stream
var lines = string.split("\n");   // split the lines

lines.reverse();                  // reverse the lines

var reversed = lines.join("\n");  // join the reversed lines
system.stdout.write(reversed);    // write the reversed lines

This allows you to fire up a shell and run the tac command. Then you type in as many lines as you wish to and after you're done you can press Ctrl+D (or Ctrl+Z on Windows) to signal the end of transmission.

I want to do the same thing in node.js but I can't find any function which would do so. I thought of using the readSync function from the fs library to simulate as follows, but to no avail:

fs.readSync(0, buffer, 0, buffer.length, null);

The file descriptor for stdin (the first argument) is 0. So it should read the data from the keyboard. Instead it gives me the following error:

Error: ESPIPE, invalid seek
    at Object.fs.readSync (fs.js:381:19)
    at repl:1:4
    at REPLServer.self.eval (repl.js:109:21)
    at rli.on.self.bufferedCmd (repl.js:258:20)
    at REPLServer.self.eval (repl.js:116:5)
    at Interface.<anonymous> (repl.js:248:12)
    at Interface.EventEmitter.emit (events.js:96:17)
    at Interface._onLine (readline.js:200:10)
    at Interface._line (readline.js:518:8)
    at Interface._ttyWrite (readline.js:736:14)

How would you synchronously collect all the data in an input text stream and return it as a string in node.js? A code example would be very helpful.

Gwenny

You can't synchronously read in an asynchronous stream. Why would you want to anyway? – Inert

I'm trying to do the same thing. The reason is to create a interactive option in my program, useful for a lot of reasons. A async reader do not help too much. – Tarah

here a way npmjs.com/package/readline-sync: #8453457 – Tarah

Answer 1

The key is to use these two Stream events:

Event: 'data'
Event: 'end'

For stream.on('data', ...) you should collect your data data into either a Buffer (if it is binary) or into a string.

For on('end', ...) you should call a callback with you completed buffer, or if you can inline it and use return using a Promises library.

Meghannmegiddo

Answer 2

As node.js is event and stream oriented there is no API to wait until end of stdin and buffer result, but it's easy to do manually

var content = '';
process.stdin.resume();
process.stdin.on('data', function(buf) { content += buf.toString(); });
process.stdin.on('end', function() {
    // your code here
    console.log(content.split('').reverse().join(''));
});

In most cases it's better not to buffer data and process incoming chunks as they arrive (using chain of already available stream parsers like xml or zlib or your own FSM parser)

Fajardo

You can do process.stdin.setEncoding('utf-8'); after resume and bug in callback will already be string. – Baum

Similar, but using Buffer.concat(): #10687117 – Altruistic

@Mitar: it's buf, not bug. – Abshier

Why reverse the string? – Renettarenew

that was just an example of doing something with data – Fajardo

Note: If nothing is passed to stdin, your program will hang. You can detect if your script is being called in a pipe using process.stdin.isTTY, as detailed in this other answer on stackoverflow – Hatch

Answer 3

Let me illustrate StreetStrider's answer.

Here is how to do it with concat-stream

var concat = require('concat-stream');

yourStream.pipe(concat(function(buf){
    // buf is a Node Buffer instance which contains the entire data in stream
    // if your stream sends textual data, use buf.toString() to get entire stream as string
    var streamContent = buf.toString();
    doSomething(streamContent);
}));

// error handling is still on stream
yourStream.on('error',function(err){
   console.error(err);
});

Please note that process.stdin is a stream.

Seng

Answer 4

There is a module for that particular task, called concat-stream.

Pontifex

This module allows you to intersperse the chunks with another string. Probably only useful for debugging: npmjs.org/package/join-stream – Altruistic

Answer 5

If you are in async context and have a recent version of Node.js, here is a quick suggestion:

const chunks = []
for await (let chunk of readable) {
  chunks.push(chunk)
}
console.log(Buffer.concat(chunks))

Sciential

Brilliant answer, short and to the point, perfect for my use case. Good stuff! – Glidden

Answer 6

This is an old question but it's worth mentioning that Node.js has some new stream helpers, one of which is toArray:

require('http')
    .createServer(async (req, res) => {
        const str = (await req.toArray()).toString().toUpperCase();
        res.end(str);
    })
    .listen(4000);

Please note: this API is currently marked as experimental so may be better suited for testing/non-production code.

Ornithine

Answer 7

On Windows, I had some problems with the other solutions posted here - the program would run indefinitely when there's no input.

Here is a TypeScript implementation for modern NodeJS, using async generators and for await - quite a bit simpler and more robust than using the old callback based APIs, and this worked on Windows:

import process from "process";

/**
 * Read everything from standard input and return a string.
 * 
 * (If there is no data available, the Promise is rejected.)
 */
export async function readInput(): Promise<string> {  
  const { stdin } = process;

  const chunks: Uint8Array[] = [];

  if (stdin.isTTY) {
    throw new Error("No input available");
  }

  for await (const chunk of stdin) {
    chunks.push(chunk);
  }

  return Buffer.concat(chunks).toString('utf8');
}

Example:

(async () => {
  const input = await readInput();

  console.log(input);
})();

(consider adding a try/catch, if you want to handle the Promise rejection and display a more user-friendly error-message when there's no input.)

Laurettalaurette