How to avoid fast memory increase during scavenge gc?

Question

I have an application built on restify. I have no memory leaks, however I have big memory growth during scavenge gc, then comes heavy weight mark-sweep gc and cleans the memory.

It affects performance of my application.

[2268]   266859 ms: Scavenge 61.5 (119.5) -> 46.0 (119.5) MB, 2.2 ms [allocation failure].
[2268]   267084 ms: Scavenge 63.7 (119.5) -> 48.3 (119.5) MB, 6.2 ms [allocation failure].
[2268]   267289 ms: Scavenge 66.0 (119.5) -> 50.6 (119.5) MB, 2.6 ms [allocation failure].
[2268]   267504 ms: Scavenge 68.3 (119.5) -> 52.8 (119.5) MB, 2.4 ms [allocation failure].
[2268]   267700 ms: Scavenge 70.5 (119.5) -> 55.1 (119.5) MB, 2.7 ms [allocation failure].
....

[2268]   275913 ms: Scavenge 154.2 (183.5) -> 138.8 (183.5) MB, 2.4 ms [allocation failure].
[2268]   276161 ms: Scavenge 157.5 (185.5) -> 142.1 (185.5) MB, 2.7 ms (+ 2.4 ms in 18 steps since last GC) [allocation failure].
[2268]   276427 ms: Scavenge 159.8 (187.5) -> 144.3 (187.5) MB, 2.5 ms (+ 36.3 ms in 236 steps since last GC) [allocation failure].
[2268]   276494 ms: Mark-sweep 147.7 (188.5) -> 43.7 (121.5) MB, 20.1 ms (+ 45.1 ms in 298 steps since start of marking, biggest step 0.5 ms) [GC interrupt] [GC in old space requested].

This type of behavior happens when I try to access non-existing url

ab -c 100 -n 10000000 -k http://localhost:1337/invalid/url

I can't really use node-inspector to track what causes such intense memory growth because it will request full gc before taking heap snapshot.

What are my options to track what causes such a rapid memory growth?

How to find out which objects survive scavenges but don't survive mark-sweep gc?

Thanks,

UPDATE 1 So there is no way to see middle aged scavenged content. Here is the hint, if you see fast memory increase during scavenge but then it suddenly drops with mark&sweep then it's mean that your code creates data in large space. Long stack traces for example. Restify generates gigantic stack traces which should be disabled in production.

Frosting

Answer 1

Best of my knowledge there is no simple way to see middle-aged content. However, if you see fast memory increase during scavenge and then sudden drop when mark&sweep takes place then this generally means that your code creates objects in large heap space. Long strings for example.

Rectify generates gigantic stack traces with each Error object which are large enough to not to fit into new space and therefore be ignored by mark&sweep.

Frosting

Answer 2

You might try running your Node script with the option -–expose-gc:

node --expose-gc script.js

This allows to trigger the garbage collection manually from within JS:

global.gc();

When garbage collection is enforced manually, you can apply a multiple snapshot technique:

• take one Snap before, one Snap after GC
• then apply optimization
• then one Snap before, one Snap after GC

The snapshots allow to track what causes the memory growth. The goal is to have a better result of the second "Snap after GC", when compared to the first "Snap after GC".

Immediacy

Sorry for dumb question, but how do I take snapshot without invoking gc? – Frosting

After executing your node application, fire up node-inspector and connect to http://127.0.0.1:8080/debug?port=5858 (or your host instead of localhost) with for instance Chrome Dev Tools and then go the "Profiles" tab. You will find "Take Heap Snapshot" and "Record Heap Allocation" there. – Immediacy

Yes, that's what I'm taking about. I see in logs that before make full snapshot it run's full gc – Frosting

Garbage Collection is only forced automatically, when taking a Heap Snapshot. In other words: "Record the Heap Allocation". It will take snapshots throughout the recording and then one final snap at the end. developers.google.com/web/tools/profile-performance/… – Immediacy

Weird, I still see only Mark-sweep gc called during Heap Allocations snapshot – Frosting

[it might be hard or impossible to capture "middle-aged" garbage with heap snapshots because V8 does full garbage collection before taking snapshot thus effectively killing all such garbage]. mrale.ph/blog/2011/12/18/v8-optimization-checklist.html – Frosting