Issue With NodeJS Memory Consumption Statistics - Huge RSS & External Memory Usage

Question

While working on performance testing for a Node Project, I came across some memory reports which are a bit confusing to interpret in terms of performance and memory leaks.

Background

So I generated the reports using process.memoryUsage() API that provides information about following -

• RSS - subset of the total allocated memory including includes the heap, code segment and stack
• heapTotal - memory allocated to the objects, strings, and closures
• heapUsed - actual memory used by the objects, strings, and closures
• external - memory usage of C++ objects bound to JavaScript objects managed by V8

---

The Problem

I recorded the memory snapshot after every request made to the server and plotted a graph for 100 requests -

The graph shows following information -

1. The RSS memory keeps increasing with every request served
2. The External Memory usage is significant compared to Heap Memory
3. The External Memory goes down once the process is idle
4. The Total Heap & Used Heap memories are well under control

---

Findings

As per my understanding, something outside Heap is causing the memory to rise.

I am using sharp node package which is resizing a lot of large images which ends up generating huge in memory buffers for those images.

In terms of memory, every request is serving 10 images that ends up generating 80 MB of buffer data

This is where I came across this github issue where impact of creating buffers in NodeJs is discussed

this answer in the same github issue, explains how buffers can possibly occupy memory that garbage collector can't keep track of

---

Questions

Am I on the right track?

If not, is using buffers in Node Applications not a proper way?

Forb

Did you find a solution to this problem? I am facing the exact same issue with sharp. Even without image processing, the RSS keeps on rising in my application. But with sharp, there is a significant rise. – Infusion

@AdityaVarma I did get a solution for it. The memory consumption still follows the same pattern. As a workaround, I am running the Node service is PM2 Cluster Mode with max memory auto reload enabled. Of course it is not the correct solution but It kind of makes sure memory usage stays in limit. – Forb

I was finally able to fix this issue by using libjemalloc as the memory allocator. For some reasons, it is better at returning the unused memory back to the OS. You can find the detailed explanation here: github.com/lovell/sharp/issues/2324 – Infusion

Do you have any native Node.js modules installed? It is up to each individual Node.js native module to track the memory it uses - and it is very easy to miss this – Drachma

Thanks @mmomtchev, I will check this out! – Forb