How do I read the contents of a Node.js stream into a string variable?

Question

How do I collect all the data from a Node.js stream into a string?

Satterlee

You should copy the stream or flag it with (autoClose: false). It is bad practice to pollute the memory. – Concentrate

Answer 1

(This answer is from years ago, when it was the best answer. There is now a better answer below this. I haven't kept up with node.js, and I cannot delete this answer because it is marked "correct on this question". If you are thinking of down clicking, what do you want me to do?)

The key is to use the data and end events of a Readable Stream. Listen to these events:

stream.on('data', (chunk) => { ... });
stream.on('end', () => { ... });

When you receive the data event, add the new chunk of data to a Buffer created to collect the data.

When you receive the end event, convert the completed Buffer into a string, if necessary. Then do what you need to do with it.

Lodged

A couple of lines of code illustrating the answer is preferable to just pointing a link at the API. Don't disagree with the answer, just don't believe it is complete enough. – Eugeniaeugenics

With newer node.js versions, this is cleaner: https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable – Toadeater

The answer should be updated to not recommend using a Promises library, but use native Promises. – Clifton

@DanDascalescu I agree with you. The problem is that I wrote this answer 7 years ago, and I haven't kept up with node.js . If you are someone else would like to update it, that would be great. Or I could simply delete it, as there seems to be a better answer already. What would you recommend? – Lodged

@ControlAltDel: I appreciate your initiative to delete an answer that is no longer the best. Wish others had similar discipline. – Clifton

@DanDascalescu Unfortunately, when I tried to delete, SO won't let me as it is the selected answer. – Lodged

Turns out the actual best answer came late to the party: https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable – Impertinent

Answer 2

Another way would be to convert the stream to a promise (refer to the example below) and use then (or await) to assign the resolved value to a variable.

function streamToString (stream) {
  const chunks = [];
  return new Promise((resolve, reject) => {
    stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
    stream.on('error', (err) => reject(err));
    stream.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
  })
}

const result = await streamToString(stream)

Unruh

I'm really new to streams and promises and I'm getting this error: SyntaxError: await is only valid in async function. What am I doing wrong? – Toluene

You have to call the streamtostring function within a async function. To avoid this you can also do streamToString(stream).then(function(response){//Do whatever you want with response}); – Parley

This should be the top answer. Congratulations on producing the only solution that gets everything right, with (1) storing the chunks as Buffers and only calling .toString("utf8") at the end, to avoid the problem of a decode failure if a chunk is split in the middle of a multibyte character; (2) actual error handling; (3) putting the code in a function, so it can be reused, not copy-pasted; (4) using Promises so the function can be await-ed on; (5) small code that doesn't drag in a million dependencies, unlike certain npm libraries; (6) ES6 syntax and modern best practices. – Delcine

Why not move the chunks array into the promise? – Nimbus

After I came up with essentially the same code using current top answer as the hint, I have noticed that above code could fail with Uncaught TypeError [ERR_INVALID_ARG_TYPE]: The "list[0]" argument must be an instance of Buffer or Uint8Array. Received type string if the stream produces string chunks instead of Buffer. Using chunks.push(Buffer.from(chunk)) should work with both string and Buffer chunks. – Deepdyed

Turns out the actual best answer came late to the party: https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable – Impertinent

chunks.push(Buffer.from(chunk)) gives a typescript error Argument type Buffer | string is not assignable to parameter type WithImplicitCoercion<ArrayBuffer | SharedArrayBuffer>. Is this a valid error or are the types broken? – Fantasy

What's the reason for this edit? I write my code more similarly to the previous revision (directly passing reject as the callback instead of calling it)--will that run into errors? – Copacetic

@Copacetic The reasoning behind the edit is that is generally considered bad practice to pass a callback to a function that was not designed for it. There was nothing wrong with the previous version, though - it should work just fine. One could argue it might break if the Promise.resolve API ever changes, but that's very unlikely to happen. I've just edited it in order to encourage what I see as best practices. – Unruh

If you attempt to read a big file (text) and do something with the data, u need to append the next chunk based on prev chunk (remaining invalid bytes) to ensure that in the case a chunk cuts off your bytes, you are re-computing them on the next read. To do that use string_decoder. Here's a usage example Edit: I'd advise streaming it line-by-line if you need to work with the file data: readLine.createInterface({ input: stream, crlfDelay: Infinity }) – Retortion

In my case I was happy with const packagePath = execSync("rospack find my_package").toString(). – Vend

Just commenting here to ensure the top answer has a reference to the new best solution, which is the NodeJS native reader: import { text } from 'node:stream/consumers'; https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable – Downtoearth

Answer 3

What do you think about this ?

async function streamToString(stream) {
    // lets have a ReadableStream as a stream variable
    const chunks = [];

    for await (const chunk of stream) {
        chunks.push(Buffer.from(chunk));
    }

    return Buffer.concat(chunks).toString("utf-8");
}

Causeway

Works, very clean, no dependencies, nice! – Ordovician

What a nice and unexpected solution! You are a Mozart of javascript streams and buffers! – Anaximander

Had to use chunks.push(Buffer.from(chunk)); to make it work with string chunks. – Vierno

Wow, this looks very neat! Does this have any problems (aside from the one mentioned in the above comment)? Can it handle errors? – Copacetic

This is the modern equivalent to the top answer. Whew Node.js/JS changes fast. I'd recommend using this one as opposed to the top rated one as it's much cleaner and doesn't make the user have to touch events. – Twombly

My IDE claims 'for await (let...' to be errornous. – Ringsmuth

@DirkSchumacher Your IDE either uses outdated script interpreter (for await is a valid ECMAScript syntax) or is itself outdated if it attempts to (unsuccessfully) execute some code containing for await. Which IDE is it? Anyway, IDEs aren't designed to actually run programs "in production", they lint them and help with analysis during development. – Backstroke

@amn I am running in the most up to date IntelliJ-Ultimate. Though it is more a Java IDE it knows somewhat web stuff as well. I am sitting behind a company firewall so this and other things may be the issue. Thanks for your advise/info/idea! – Ringsmuth

@DirkSchumacher No bother. Just see if you can find out exactly what component of your IDE -- I assume it will be a program -- loads and executes the script containing for await. Query the version of the program and find out if the version actually supports the syntax. Then find out why your IDE is using the particular "outdated" version of the program and find a way to update both. – Backstroke

FYI, had to push the chunk using Buffer.from, ex. chunks.push(Buffer.from(chunk)), but otherwise a nice and elegant solution! thanks! – Paule

Answer 4

None of the above worked for me. I needed to use the Buffer object:

  const chunks = [];

  readStream.on("data", function (chunk) {
    chunks.push(chunk);
  });

  // Send the buffer or you can put it into a var
  readStream.on("end", function () {
    res.send(Buffer.concat(chunks));
  });

Coursing

this is actually the cleanest way of doing it ;) – Stitch

Works great. Just a note: if you want a proper string type, you will need to call .toString() on the resulting Buffer object from concat() call – Extemporize

Turns out the actual best answer came late to the party: https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable – Impertinent

this is the only way to do it correctly – Juggler

Answer 5

Hope this is more useful than the above answer:

var string = '';
stream.on('data',function(data){
  string += data.toString();
  console.log('stream data ' + part);
});

stream.on('end',function(){
  console.log('final output ' + string);
});

Note that string concatenation is not the most efficient way to collect the string parts, but it is used for simplicity (and perhaps your code does not care about efficiency).

Also, this code may produce unpredictable failures for non-ASCII text (it assumes that every character fits in a byte), but perhaps you do not care about that, either.

Prelacy

What would be a more efficient way to collect string parts? TY – Kraken

you could use a buffer docs.nodejitsu.com/articles/advanced/buffers/how-to-use-buffers but it really depends on your use. – Prelacy

Use an array of strings where you append each new chunk to the array and call join("") on the array at the end. – Ginger

This isn't right. If buffer is halfway through a multi-byte code point then the toString() will receive malformed utf-8 and you will end up with a bunch of � in your string. – Magee

i'm not sure what you mean by a 'mutli-byte code point' but if you want to convert the encoding of the stream you can pass an encoding parameter like this toString('utf8') - but by default string encoding is utf8 so i suspect that your stream may not be utf8 @Magee - see #12122275 for more – Prelacy

@Magee is right. In some very rare cases when I had a lot of chunks I got those � at the start and end of chunks. Especially when there where russian symbols on the edges. So it's correct to concat chunks and convert them on end instead of converting chunks and concatenating them. In my case the request was made from one service to another with request.js with default encoding – Fredrick

Answer 6

I'm using usually this simple function to transform a stream into a string:

function streamToString(stream, cb) {
  const chunks = [];
  stream.on('data', (chunk) => {
    chunks.push(chunk.toString());
  });
  stream.on('end', () => {
    cb(chunks.join(''));
  });
}

Usage example:

let stream = fs.createReadStream('./myFile.foo');
streamToString(stream, (data) => {
  console.log(data);  // data is now my string variable
});

Goosander

Useful answer but it looks like each chunk must be converted to a string before it is pushed in the array: chunks.push(chunk.toString()); – Icelandic

This is the only one that worked for me ! Great thanks – Isiah

This was a great answer! – Matronymic

There is an edge case here when a multi-byte character lands is split between chunks. This results in the original character being replaced with two incorrect characters. – Sexual

@NicolasLeThierryd'Ennequin chunk are Buffer so you can concatenate them. – Corbel

Answer 7

And yet another one for strings using promises:

function getStream(stream) {
  return new Promise(resolve => {
    const chunks = [];

    # Buffer.from is required if chunk is a String, see comments
    stream.on("data", chunk => chunks.push(Buffer.from(chunk)));
    stream.on("end", () => resolve(Buffer.concat(chunks).toString()));
  });
}

Usage:

const stream = fs.createReadStream(__filename);
getStream(stream).then(r=>console.log(r));

remove the .toString() to use with binary Data if required.

update: @AndreiLED correctly pointed out this has problems with strings. I couldn't get a stream returning strings with the version of node I have, but the api notes this is possible.

Careycarfare

I have noticed that above code could fail with Uncaught TypeError [ERR_INVALID_ARG_TYPE]: The "list[0]" argument must be an instance of Buffer or Uint8Array. Received type string if the stream produces string chunks instead of Buffer. Using chunks.push(Buffer.from(chunk)) should work with both string and Buffer chunks. – Deepdyed

Answer 8

It's easiest using Node.js built-in streamConsumers.text:

import { text } from 'node:stream/consumers';
import { Readable } from 'node:stream';

const readable = Readable.from('Hello world from consumers!');
const string = await text(readable);

Grassgreen

Answer 9

Easy way with the popular (over 5m weekly downloads) and lightweight get-stream library:

https://www.npmjs.com/package/get-stream

const fs = require('fs');
const getStream = require('get-stream');

(async () => {
    const stream = fs.createReadStream('unicorn.txt');
    console.log(await getStream(stream)); //output is string
})();

Rafi

Answer 10

From the nodejs documentation you should do this - always remember a string without knowing the encoding is just a bunch of bytes:

var readable = getReadableStreamSomehow();
readable.setEncoding('utf8');
readable.on('data', function(chunk) {
  assert.equal(typeof chunk, 'string');
  console.log('got %d characters of string data', chunk.length);
})

Sutra

Answer 11

Streams don't have a simple .toString() function (which I understand) nor something like a .toStringAsync(cb) function (which I don't understand).

So I created my own helper function:

var streamToString = function(stream, callback) {
  var str = '';
  stream.on('data', function(chunk) {
    str += chunk;
  });
  stream.on('end', function() {
    callback(str);
  });
}

// how to use:
streamToString(myStream, function(myStr) {
  console.log(myStr);
});

Louanne

Answer 12

I had more luck using like that :

let string = '';
readstream
    .on('data', (buf) => string += buf.toString())
    .on('end', () => console.log(string));

I use node v9.11.1 and the readstream is the response from a http.get callback.

Strafe

Answer 13

Even if this answer was made 10 years ago, I consider it's important to add my answer since there are a couple of popular answers that do not consider the official docs of Node.js (https://nodejs.org/api/stream.html#readablesetencodingencoding) which say:

The Readable stream will properly handle multi-byte characters delivered through the stream that would otherwise become improperly decoded if simply pulled from the stream as Buffer objects.

That's the reason I will modify the two most popular answers showing the best way to do the encoding process:

function streamToString(stream) {
    stream.setEncoding('utf-8'); // do this instead of directly converting the string
    const chunks = [];
    return new Promise((resolve, reject) => {
        stream.on('data', (chunk) => chunks.push(chunk));
        stream.on('error', (err) => reject(err));
        stream.on('end', () => resolve(chunks.join("")));
    })
}

const result = await streamToString(stream)

or:

async function streamToString(stream) {
    stream.setEncoding('utf-8'); // do this instead of directly converting the string
    // input must be stream with readable property
    const chunks = [];

    for await (const chunk of stream) {
        chunks.push(chunk);
    }

    return chunks.join("");
}

Sarad

I don't see how that issue could be applicable if you concat the buffers at the end, and only then convert it to a string. Did you actually test this? – Cynewulf

Answer 14

What about something like a stream reducer ?

Here is an example using ES6 classes how to use one.

var stream = require('stream')

class StreamReducer extends stream.Writable {
  constructor(chunkReducer, initialvalue, cb) {
    super();
    this.reducer = chunkReducer;
    this.accumulator = initialvalue;
    this.cb = cb;
  }
  _write(chunk, enc, next) {
    this.accumulator = this.reducer(this.accumulator, chunk);
    next();
  }
  end() {
    this.cb(null, this.accumulator)
  }
}

// just a test stream
class EmitterStream extends stream.Readable {
  constructor(chunks) {
    super();
    this.chunks = chunks;
  }
  _read() {
    this.chunks.forEach(function (chunk) { 
        this.push(chunk);
    }.bind(this));
    this.push(null);
  }
}

// just transform the strings into buffer as we would get from fs stream or http request stream
(new EmitterStream(
  ["hello ", "world !"]
  .map(function(str) {
     return Buffer.from(str, 'utf8');
  })
)).pipe(new StreamReducer(
  function (acc, v) {
    acc.push(v);
    return acc;
  },
  [],
  function(err, chunks) {
    console.log(Buffer.concat(chunks).toString('utf8'));
  })
);

Discus

Answer 15

The cleanest solution may be to use the "string-stream" package, which converts a stream to a string with a promise.

const streamString = require('stream-string')

streamString(myStream).then(string_variable => {
    // myStream was converted to a string, and that string is stored in string_variable
    console.log(string_variable)

}).catch(err => {
     // myStream emitted an error event (err), so the promise from stream-string was rejected
    throw err
})

Kwangchow

Answer 16

All the answers listed appear to open the Readable Stream in flowing mode which is not the default in NodeJS and can have limitations since it lacks backpressure support that NodeJS provides in Paused Readable Stream Mode. Here is an implementation using Just Buffers, Native Stream and Native Stream Transforms and support for Object Mode

import {Transform} from 'stream';

let buffer =null;    

function objectifyStream() {
    return new Transform({
        objectMode: true,
        transform: function(chunk, encoding, next) {

            if (!buffer) {
                buffer = Buffer.from([...chunk]);
            } else {
                buffer = Buffer.from([...buffer, ...chunk]);
            }
            next(null, buffer);
        }
    });
}

process.stdin.pipe(objectifyStream()).process.stdout

Melva

Answer 17

If your stream doesn't have methods like .on( and .setEncoding( then you have what the newer "web fetch standards-based" ReadableStream: https://github.com/nodejs/undici/blob/c83b084879fa0bb8e0469d31ec61428ac68160d5/README.md#responsebody

You can simply do this:

const str = await new Response(request.body).text();

(I 100% just copied this from another similar SO question: https://mcmap.net/q/86735/-retrieve-data-from-a-readablestream-object)

The rest of my answer here just provides additional context on my situation, which may help catch certain keywords in google.

=====

I'm trying to convert a request.body (ReadableStream) in a SolidStart api route handler to a string: https://start.solidjs.com/core-concepts/api-routes

SolidStart takes a very isomorphic approach.

When I attempted to use the code in https://mcmap.net/q/116452/-how-do-i-read-the-contents-of-a-node-js-stream-into-a-string-variable, I got this error:

Type 'ReadableStream<any>' must have a '[Symbol.asyncIterator]()' method that returns an async iterator.ts(2504)

SolidStart uses undici as it's isomorphic fetch implementation, and newer versions of node have basically used this library for node's fetch implementation. I will be the first to admit, this doesn't seem like a "node.js stream" but in some sense it is, it's just a newer ReadableStream standard that comes with node's native fetch implementation.

For more insight, these are the methods/properties available on this request.body:

Marienbad

Answer 18

setEncoding('utf8');

Well done Sebastian J above.

I had the "buffer problem" with a few lines of test code I had, and added the encoding information and it solved it, see below.

Demonstrate the problem

software

// process.stdin.setEncoding('utf8');
process.stdin.on('data', (data) => {
    console.log(typeof(data), data);
});

input

hello world

output

object <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64 0d 0a>

Demonstrate the solution

software

process.stdin.setEncoding('utf8'); // <- Activate!
process.stdin.on('data', (data) => {
    console.log(typeof(data), data);
});

input

hello world

output

string hello world

Squabble

Answer 19

This worked for me and is based on Node v6.7.0 docs:

let output = '';
stream.on('readable', function() {
    let read = stream.read();
    if (read !== null) {
        // New stream data is available
        output += read.toString();
    } else {
        // Stream is now finished when read is null.
        // You can callback here e.g.:
        callback(null, output);
    }
});

stream.on('error', function(err) {
  callback(err, null);
})

Teutonize

Answer 20

Using the quite popular stream-buffers package which you probably already have in your project dependencies, this is pretty straightforward:

// imports
const { WritableStreamBuffer } = require('stream-buffers');
const { promisify } = require('util');
const { createReadStream } = require('fs');
const pipeline = promisify(require('stream').pipeline);

// sample stream
let stream = createReadStream('/etc/hosts');

// pipeline the stream into a buffer, and print the contents when done
let buf = new WritableStreamBuffer();
pipeline(stream, buf).then(() => console.log(buf.getContents().toString()));

Alda

Answer 21

In my case, the content type response headers was Content-Type: text/plain. So, I've read the data from Buffer like:

let data = [];
stream.on('data', (chunk) => {
 console.log(Buffer.from(chunk).toString())
 data.push(Buffer.from(chunk).toString())
});

Cleaning