Unknown objects in Managed Heap .Net 4.5

Question

I am having trouble on a server class windows 2012 R2 machine that is occasionally growing its managed .net heap to over 10GB (typically should be 1-2 GB). I have taken memory dumps on this box after a Gen2 collection to try to find which objects are getting allocated on the heap.

Using WinDbg / SOS

!dumpheap -stat

produces a list of objects, the biggest of which is ~150mb of memory, and trails down quickly. Pulling this output into excel and adding up the TotalSize, I only get 1.5 GB of objects (inline with what I expect).

              MT    Count    TotalSize Class Name
00007ffb1ab87a70        1           24 System.Collections.Generic.GenericEqualityComparer`1[[System.UInt64, mscorlib]]
...
00007ffabc178e40   218274     36670032 Ten.RunBar
00007ffabbd172f8  1048576     41943040 Ten.DisruptorTaskTuple
00007ffb1a0502c0   442990     71357200 System.Object[]
00007ffabc1bbce0   676327     97391088 Ten.BalTuple
00007ffabbd5eea8  2342912    149946368 Ten.Run.TaskAction
0000001bf119dc70  1072558    787427166      Free
MANUALLY CALCULATED TOTAL:  1589298584

My first thought is that there is some unmanaged memory leak so I checked the heap with

!EEHeap -gc

which showed that the the GC heap size itself is in fact 10GB

Number of GC Heaps: 1
generation 0 starts at 0x0000001e10623cb0
generation 1 starts at 0x0000001e10485bb0
generation 2 starts at 0x0000001b80001000
ephemeral segment allocation context: none
 segment     begin allocated  size
0000001b80000000  0000001b80001000  0000001b90000000  0xffff000(268431360)
0000001b98000000  0000001b98001000  0000001ba8000000  0xffff000(268431360)
...
0000001e04ee0000  0000001e04ee1000  0000001e109c2cc0  0xbae1cc0(195960000)
Large object heap starts at 0x0000001b90001000
 segment     begin allocated  size
0000001b90000000  0000001b90001000  0000001b9356cbc8  0x356bbc8(56015816)
Total Size:              Size: 0x25f028618 (10183935512) bytes.
------------------------------
GC Heap Size:            Size: 0x25f028618 (10183935512) bytes.

I have confirmed that the memory is all in the 2nd Gen heap (Performance Counter), but am unsure where to look for the missing 8.5 GB of managed objects from here.

Any other ideas / WinDbg / SOS commands to help find these?

Mishap

Look for pinned objects that prevent a segment from getting recycled and a deadlocked finalizer thread. – Accused

blogs.msdn.com/b/ricom/archive/2004/12/10/279612.aspx, Tracking down managed memory leaks with windbg, I used to follow this post to track .NET memory leaks, hope it helps. – Metzgar

I've used his site for tracking down leaks, helpful, but inconclusive for my issue. I will check the pinned objects, though I know I use ~10-15 statically pinned objects, which should not be able to cause this. – Mishap

Windows Sysinternals VMMap can show you some memory areas occupied by managed heap. It may give you some hint as for where to look at and what is also in the memory that might influence the managed heap – Sandpaper

Answer 1

The sizes reported by !dumpheap are the sizes of the types themselves and not the actual sizes of the instances.

You can use !objsize to list the size of an instance. However, as multiple instance might share references the same objects summing the output of !objsize usually creates an incorrect result as well.

The best way to figure out the size of the managed heap is to look at the totals reported by !dumpheap and !eeheap.

Hydromedusa

the sizes in dump heap should still sum up to the total managed memory heap though, the children of the objects will be included higher up on the list. Where could this uncounted memory be from? – Mishap

I see what you mean. The number reported by !eeheap is the size used by the CLR to managed the managed heap. It is not the sum of managed objects. Instead it is is the sum of the segments allocated to store the objects. This number can be significantly larger than the space used by managed objects on the heap. As Hans points out in his comment to your question, certain situations can prevent the CLR from releasing segments which results in a large difference between memory allocated for the managed heap and the memory needed to store the objects on the heap. – Hydromedusa