Recommended way to grow a Buffer?

Question

Let's say I am constructing a string, or series of bytes, of variable length, in Node. The documentation for buf.write says:

https://nodejs.org/api/buffer.html#buffer_buf_write_string_offset_length_encoding

Writes string to buf at offset according to the character encoding in encoding. The length parameter is the number of bytes to write. If buf did not contain enough space to fit the entire string, only a partial amount of string will be written. However, partially encoded characters will not be written.

Let's say I want to write more data than the Buffer has room for. What is the recommended way to grow it? It doesn't seem like there is a .grow method or similar. I can call Buffer.from(buf) to create a new Buffer, but that seems inefficient.

Kinsman

As I recall, ArrayLists that are backed by arrays deal with too many items by producing another array with double the size. It is trash for when you have to keep resizing, but it works great when you know about how big it will be. – Torture

Answer 1

In case if you don't need all the contents of the whole buffer after every addition I think you can use an array of chunks (buffers) here and Buffer.concat(chunks) to build the whole buffer when necessary.

const chunks = [];

chunks.push(Buffer.from([0xE2, 0xAD, 0x90]));  // star
chunks.push(Buffer.from([0xF0, 0x9F, 0x9A, 0x80])); // rocket

const buf = Buffer.concat(chunks);

console.log(buf.toString()); 

Degrade

Answer 2

When the final size is not known at the beginning, you can use the approach typically applied for dynamic arrays:

1. allocate Buffer with some initial size
2. write content to Buffer as long as there is space
3. when the next piece of content cannot fit, create a new Buffer with double the size, copy old buffer's content to the new buffer, and write the new content to the end as before

That way inserting at the end of the buffer can be performed in O(1) amortized time.

Variants of this approach are commonly used in resizable lists in Java, .NET, or e.g. in .NET's MemoryStream which corresponds to nodejs Buffer. You can see how MemoryStream is implemented - here.

Size does not necessarily have to be doubled. You can choose a different exponent, e.g. 3/2, if you want to save some space.

Of course what is the best approach depends on your use case, but this one performs well in general.

Mlawsky

You mean "Write content to Buffer as long as there is space"? – Fleshpots

@DavidsKanal, yes I've corrected my answer. Thanks. – Mlawsky

Awesome! Happy to help. – Fleshpots

Answer 3

There is no way to change length of existing Buffer

Instances of the Buffer class are similar to arrays of integers but correspond to fixed-sized, raw memory allocations outside the V8 heap. The size of the Buffer is established when it is created and cannot be resized.

https://nodejs.org/api/buffer.html#buffer_buffer

However you can try to use/write some other buffer implementation or use Buffer.concat as Nik Markin suggests

Warga

Answer 4

I am very new to Node JS, but when I tried following code it worked.
[Not sure if it creates any memory leaks or if there are any further issues...]

'use strict';

var tempBuffer = [];

var i = 0;

tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);

console.log('buf4 before creation: ');
console.log(buf4);

var buf4 = Buffer.from(tempBuffer);
console.log('buf4 after creation (type):' + typeof(buf4));

console.log('buf4 after creation (data):');
console.log(buf4);

tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);
tempBuffer.push(i++);

buf4 = Buffer.from(tempBuffer);

console.log('buf4 after expantion (data):');
console.log(buf4);

And here is output:

buf4 before creation:
undefined
buf4 after creation (type):object
buf4 after creation (data):
<Buffer 00 01 02 03 04 05>
buf4 after expantion (data):
<Buffer 00 01 02 03 04 05 06 07 08 09 0a 0b>

Petiole

You literally just allocated a new Buffer (initialized from the same array, after its modification) and assigned it to the same variable name. At no point did you modify the first buffer you allocated. – Emphasize

Answer 5

I tried Nik's answer which works nicely but on some large Buffers (large image bitmaps) I was running out of heap and getting errors!

I had better luck with:

// create instance of big enough buffer
const biggerBuffer = Buffer.alloc(smallerBufferLength + diffLen);
// copy stuff across
const count = smallerBuffer.copy(biggerBuffer);
// allow old buffer to be GC'ed
smallerBuffer = null;

The remaining space is by default filled with zeros: https://nodejs.org/api/buffer.html#static-method-bufferallocsize-fill-encoding.

:o)

Folie