NodeJS ReadStream not reading bufferSize bytes at a time

Question

I have a code where the NodeJS server reads a file and streams it to response, it looks like:

var fStream = fs.createReadStream(filePath, {'bufferSize': 128 * 1024});
fStream.pipe(response);

The issue is, Node reads the file exactly 40960 bytes a time. However, my app would be much more efficient (due to reasons not applicable to this question), if it reads 131072 (128 * 1024) bytes at a time.

Is there a way to force Node to read 128 * 1024 bytes at a time?

Moina

Answer 1

I'm new here, so bear with me....

I found this in node's sources:

var toRead = Math.min(pool.length - pool.used, ~~this.bufferSize);

and:

var kPoolSize = 40 * 1024;

So it seems that the buffer size is limited to 40kb, no matter what you provide. You could try to change the value in the code and rebuild node. That's probably not a very maintainable solution though...

When

This is right. I edited the value to 128 * 1024 and rebuilt the node. It still doesn't do the job, but its still better. It usually reads about 128000 bytes in one read, and then the remaining balance (128k - ~128000) in the next read. Which is kinda strange. – Moina

Answer 2

The accepted answer is wrong. You can force Node to read (128*1024) bytes at a time using the highWaterMark option.

var fStream = fs.createReadStream('/foo/bar', { highWaterMark: 128 * 1024 });

The Documentation specifically states that 'the amount of data potentially buffered depends on the highWaterMark option passed into the streams constructor. For normal streams, the highWaterMark option specifies a total number of bytes. For streams operating in object mode, the highWaterMark specifies a total number of objects.'

Also, see this. The default buffer size is 64 KiloBytes

Rociorock