Analysing a crash dump on Windows, most of the memory is "busy internal", what is it?

Question

I'm running a 32bit application and it fails on shortage of memory (bad_alloc is thrown). However, when I'm looking in the dump file I can see that most of the memory is in "busy internal". Moreover, it has 3GB of committed memory. To put it as questions

1. Does it make sense that an app has so much "busy internal" memory?
2. Is it something I can control and/or configure?
3. Can I trigger some sort of memory consolidation of defragmentation?

DebugDiag:

Virtual Memory Analysis 
Virtual Memory Summary 
Size of largest free VM block   6.91 MBytes 
Free memory fragmentation       92.S6% 
Free Memory                     96.78 MBytes (2.36% ofT0ta1 Memory) 
Reserved Memory                 856.05 MBytes (20.9% ofT0ta1 Memory)
Committed Memory                3.07 GBytes (76.7400 ofT0ta1 Memory)
Total Memory                    4 GBytes 
Largest free block at           0x00000000'ff1f6000 

Partial output of WinDBG (!heap -a <heap>)

e3268eee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e32aaee8:  42008 . 21ff8 [101] - busy (21ff0) Internal 
e32cceee:  21ff8 . 22008 [101] - busy (21ff8) Internal 
e32eeee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e331eeee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e3352008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3394eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 
e33b6008:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e33d8eee:  21ff8 . 22008 [101] - busy (21ff8) Internal 
e33faee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e341ceee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e345eee8:  42008 . 21ff8 [101] - busy (21ff0) Internal 
e348eeee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e34c2008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3504eee:  41ff8 . 42008 [101] - busy (41ff8) Internal 
e3546008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3588eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 
e35aaee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e35cceee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e36eeee8:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e365eeee:  41ff8 . 42008 [101] - busy (41ff8) Internal 
e3692008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e36d4eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 

UPDATE

Adding some more info in this - finding the total size of busy-internal

************************************************************************************************************************
                                              NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key                   : 0xc6ab7f8e
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
01590000 00000002 2995168 2981356 2994956  27250  2871   211    4   1db2   LFH
01bb0000 00001002    1292    536   1080     46     5     2    0      0   LFH
-----------------------------------------------------------------------------

I ran !heap -a 01590000 and saved the output to a log. Then I summed up the results, categorized by 'free', 'busy', 'busy-internal' and these were my results

busy 1.106 MB
busy-internal 2883.606 MB
free 26.612 MB
uncategorized 26.612 MB
total bytes 3080648904

This result agrees with the size of the memory shows, for both the total and free memory. It is clear that most of the memory is marked as "internal" for some vague reason.

Live

You can find many useful guides from Tess tessferrandez.com/postindex and other authors on how to analyze high memory dumps. But a 32 bit process dump of more than 3 Gb memory usage already tells a lot, as 4-Gb is the limit. – Bowerman

Answer 1

Busy vs. free

A heap by the Windows Heap Manager is divided in segments and those segments are divided in blocks. Each block can basically have two states: free and busy, where busy means "in use". Assuming C++, int* pi = new int; gives you a busy block and delete pi; turns it back into a free block.

Does it make sense that an app has so much "busy internal" memory

"An app" - we don't know. Your app? Maybe. It depends on how often a new statement is invoked.

Is it something I can control and/or configure?

Sure. But you don't want to and you shouldn't. You can:

1. try to get rid of all new statements in your application. Question is: where do you want to store your data instead? Typically, you can exchange memory usage against CPU cycles and/or disk operations. E.g. you could write all data into a file and only read that portion from the file which is currently needed.
2. call VirtualAlloc() and then managing the memory yourself. This will reduce busy blocks, because you bypass the Windows Heap Manager completely. However, you would need to implement your own Heap Manager, which may result in something identical to busy blocks.
3. Reduce the overall available memory using GFlags. That way your app will fail earlier but you will have less busy blocks. Not helpful, I just took your request literally.

Can I trigger some sort of memory consolidation of defragmentation?

No. The Windows Heap Manager is sorting blocks by size already and reusing regions of the same size for the next new statement of the same size.

What you should consider:

• check your app for memory leaks. If this much memory is unexpected, then a memory leak can explain it. Try UMDH or a commercial tool, Valgrind if your app also runs on Linux.
• compile your app as 64 bit. You will have so much more memory available.
• if you compile your app in debug mode, switch to release mode. Release builds will use a different Heap Manager than debug builds. Also, optimization may reduce memory usage.
• if you make heavy use of the PImpl idiom, switching from arrays of pointers to arrays of data may help (also with performance).

Internal

As @Neitsa has once reverse-engineered, the Internal output corresponds to the HEAP_ENTRY_VIRTUAL_ALLOC flag.

IMHO, this means that the heap manager has decided to not manage this allocation and has forwarded the allocation to VirtualAlloc() instead. One consequence of this is that you won't have user mode stack traces for those allocations.

Ansley

I understand that busy is memory in use, however what about the "internal" indication? what does it mean that over 50% of the memory is internal? – Live

As described in the answer, Internal means that it's a memory block that the heap manager decided to just allocate by VirtualAlloc (because it's so big it's not worth mixing in with the other heap manager blocks). Notice that all of your "Internal" blocks are very close to multiples of 4KB in size. Each one represents a large memory allocation that hasn't yet been freed. – Felic

@RaymondChen: Mario Hewardt explains in "Advanced Windows Debugging" on page 264 that the limit should be ~512k (0x7FF0). Why does OP see smaller sizes like ~128k and ~256k here? Sure, my book is 8th edition from 2012, which is some time ago. Is it related to the latest variable size heaps? – Ansley

@ThomasWeller Okay, I looked more closely, and the Internal does not mean "large allocation". It's an internal flag used by the heap manager (also numerical value 8, which may be what confused Neitsa's analysis) for memory blocks used for internal bookkeeping. (e.g., keeping track of which blocks are free and which are busy). – Felic

@RaymondChen: in that case, I'd delete my answer and beg you to write an answer instead. You are a more credible source for such a statement. – Ansley

Unfortunately, I don't have an answer. – Felic

That was my first question - how come that 3GB are needed for internal book keeping (of x86 app)? – Live

@tamir: there's no evidence in your question that 3GB are "Internal". From the output we see only 4.7 MB. – Ansley

I summed all the sizes of blocks that were marked as busy internal (I updated the above question). It seems that most of this app memory is busy internal. – Live

@tamir: okay, thanks – Ansley