Find memory leaks with WinDbg when lots of objects are present in Gen2

Question

I've got a memory problem in my .NET application where my app starts off consuming about 1GB in the Gen2 heap after everything is initialized and loaded. It slowly over time (4-5 hours) ends up consuming 4GB in the Gen2 heap. I've used WinDbg to analyze things that I see that some of my object types (and associated memory usage) are increasing.

All of the objects that growing in instances (and mem usage) are referenced by the same parent object type. This parent object type has about 3900 instances - this never changes. Somehow, I'm adding child objects to some of these parent instances, but I don't have a good way to see which of the 3900 instances are being added to.

!DumpHeap -mt will show me all my parent types, but the sizes are all the same because it doesn't count children.

!ObjSize will count the size of the children too, but will only take one object at a time for an argument (or all objects of all types - not just my parent type - which is way too many objects)

Looking at the child objects and tracing them back to the parent isn't helpful either because there are a couple million of these types and I don't see a way to do some kind of aggregate trace.

Tools like CLRProfiler and ANTS slow down my app too much (ANTS less so) to make the problem happen in any reasonable amount of time.

I have tried running my application with a small subset of data that it normally runs on in order to make debugging easier, but I don't get memory issues here. I think there are some edge cases in my entire data set that cause strange things, but I don't know what those edge cases are in order to isolate them into a subset of my entire data set.

Have read widely on this and can't see anyone suggesting what to do when are LOTS of objects in Gen2 that are supposed to be there and a small amount of objects of the same type that keep increasing.

Any tips would be most appreciated.

Sap

Answer 1

Interesting puzzle. As you know, objects get promoted to gen2 when they've survived collections because they're referenced by something that lives a long time. You don't say what kind of application this is - asp.net, WPF, winforms, etc. so we'll have to make some guesses.

One strategy you could try is logging. You say there are 3900 instances of a 'parent object' that something is getting added to - can you instrument the method on the parent object that accepts the new objects? Maybe by logging those additions you can get an idea of where they're coming from.

Hinds

Thanks for the feedback @n8wrl. This is a Windows service, though it also compiles down to a straight console application that I run from command line. I started down the road of debugging the places where my parent object is getting added to, but VS2010 uses an implicit "remote debug" to debug 64-bit apps since VS is only 32-bit. The remote debugger keeps crashing before my memory issue happens (a common problem, Google tells me). I'm now in the process of writing messages to the console. Was thinking that there must be a better way with WinDb, perhaps even something other than SOS. – Sap

Answer 2

I assume you're using SOS for this. A better option would be to use PSSCOR2 or PSSCOR4 (depending on which version of the runtime you're on). The PSSCOR version of !dumpheap has an additional delta column, which might help you detect which instances are growing over time.

Jodhpur

Thanks Brian. I am using SOS, and I've recently checked out PSSCOR4 (although most of the new options are for ASP.NET) hoping it would help. !DumpHeap does have the change column, but the parent objects aren't growing according to !DumpHeap because !DumpHeap doesn't count children in the size. – Sap

@Dave: While PSSCOR does have a lot of ASP specific commands it is well worth getting even for just the updated versions of the SOS commands IMO. Anyway, I assume the children are heap allocated as well, so you should be able to detect an increase there. – Jodhpur

Indeed the PSSCOR additions are useful, just not sure they help in this case. The children are heap allocated, but !DumpHeap doesn't count children in the reported sizes of the parents, only !ObjSize does. I can't simply dump the children and look at their sizes either because the child sizes aren't growing, the child instances are. Dumping 2 million instances and then tracing each one back to find out which parent is truly growing in the number of instances that they hold difficult to say least! Not sure if I'm missing something in what you're suggesting though. Let me know. – Sap

@DaveRobinson: I'm not sure I follow the exact nature of how these types are used. If you're not sure where children are created, you could set breakpoints on the relevant constructors and automatically dump the call stack. You should probably just log the output to a file so you can search it afterwards. – Jodhpur

Do you know if there will be a version of PSSCOR for .NET 4.5/4.6? PSSCOR4 only works with .NET 4.0. Could it be made open source if Microsoft is not interestes in maintaining it any longer? – Caco

@Thomas I haven't seen anything yet but it would be great to get an updated version. – Jodhpur

Answer 3

You can use !objsize on all objects of your parent type

.foreach (address {!dumpheap -short -type MyParentType}) {!objsize ${address}}

or by method table if your class name is not unique enough

 !name2ee MyModule MyParentType ; *** to get the method table
 .foreach (address {!dumpheap -short -mt <methodtable>}) {!objsize ${address}}

Caco