Unmanaged memory leak in mscorwks help analyze

Question

Story:

We are facing unmanaged memory leak in our .NET 2.0 application. Process after start consume about 150MB (most of it is .NET managed, object states and so). After about 12 hours of running process consumed up to 800MB and after next 12 hours process have about 1.8GB of RAM. I just tried JetBrains .NET Memory Profiler, ANTS, .NET Memory Profiler (and maybe 2 next mem profiles available on market) none of this helps me because as i detect later our process consume that much memory in unmanaged area not managed. To detect this i used Perf monitor with counters: Private Bytes(Process) and # Bytes in All Heaps (.NET CLR Memory) where Private Bytes consume about 90% of all memory allocated by process. This is why i switch to unmanaged debugging.

DebugDiag: So i run debugdiag on process and get full dump, here is snapshot of it:

• mscorwks.dll (a known Windows memory manager) is responsible for 781,73 MBytes worth of outstanding allocations. These allocations appear to have originated from the following module(s) and function(s):

• ntdll.dll (a known Windows memory manager) is responsible for 98,24 MBytes worth of outstanding allocations. These allocations appear to have originated from the following module(s) and function(s):

Top 4 functions by allocation count

• mscorwks!EEHeapAlloc+15b -- 80 957 allocation(s)
• mscorwks!CLRMapViewOfFileEx+4a -- 4 171 allocation(s)

Top 4 functions by allocation size

• mscorwks!EEVirtualAlloc+15b -- 117,50 MBytes
• mscorwks!EEHeapAlloc+15b -- 15,03 MBytes

Interesting logs found:

Function details

Function mscorwks!EEVirtualAlloc+15b

• Allocation type Virtual memory allocation(s)
• Allocation Count 1471 allocation(s)
• Allocation Size 117,50 MBytes
• Leak Probability 73%

Function mscorwks!EEHeapAlloc+15b

• Allocation type Heap allocation(s)
• Allocation Count 80957 allocation(s)
• Allocation Size 15,03 MBytes
• Leak Probability 72%

Function mscorwks!CExecutionEngine::CheckThreadState+fe

• Allocation type Heap allocation(s)
• Heap handle 0x00000000`00000000
• Allocation Count 2 allocation(s)
• Allocation Size 304 Bytes
• Leak Probability 98%

Function mscorwks!CLRMapViewOfFileEx+4a

• Allocation type Virtual memory allocation(s)
• Allocation Count 4171 allocation(s)
• Allocation Size 0 Bytes
• Leak Probability 73%

I want someone to push me in right direction how can i found memory leak from this dump? I'm able to loaded dump into windbg and run standard set of windbg command but i dont know which one are the right commands to be able to isolate leak.

I can provide full dump if anyone want to help with this.

Savadove

Some potentially useful info here? blogs.msdn.com/b/tess/archive/2009/10/09/… Are you using any COM objects as part of your code that you might not be cleaning up? – Choirmaster

Joe: Already reading this post among with others 20+ and still dont know where & how to found leak. I have detailed log from DebugDiag but its useless when i havent clue how to 'read' it :-| – Savadove

I recommend you avoid focusing on the unmanaged side unless your app does major unmanaged memory manipulation. In a .NET app, the main cause of memory issues I have seen is by far unexpected references - event handlers not being removed being the primary cause. More in an answer below. – Juncaceous

Private bytes means memory not shared with other processes. The managed heap is not shared among processes, so if you have a large managed heap, private bytes will grow correspondingly. From your description it isn't clear to me if you see a large number for private bytes and a small number for # bytes in all heaps. If both numbers are big, you're looking at managed memory. – Huntsman

i'll give it a try. could you upload the dump somewhere and share the link? – Inflation

Answer 1

Looking at the dump file it does seem to be a managed leak, only not on the managed heap. The dump shows that the managed heap is quite small, but the loader heap is 1 GB. The process has over 35000 dynamic assemblies. I looked at a few of them and they appear to be serialization (XML and binary). Take a look at this blog post. It describes a similar issue.

Huntsman

That was the case. Deserialization of objects using Xml serializer always generate new dynamic assembly (by .NET FW itself) and after some hours there was about 35k of assemblies. So just to handle that count of assemblies consume >1GB of RAM. Thanks guys for helping isolate this "leak", where this wasn't standard leak. – Savadove

Answer 2

My favorite way to debug .NET and Silverlight memory leaks is using the SOS extension. See here for a quick walkthrough: http://blogs.msdn.com/b/ricom/archive/2004/12/10/279612.aspx

What I generally do is:

1. !DumpHeap -stat to get a list of what objects exist in memory; this will usually indicate the source of the issue (e.g. if I see a million tiny objects that should only be used once and thrown away).
2. Once I know what type of object causes it, I dump a list of such objects using !DumpHeap and randomly get the roots of a few of them (!GCRoot). This will usually indicate what object is unexpectedly holding references to the leaky objects.

This assumes that you are dealing with a managed memory leak (referenced being kept where you do not expect it). It will not help much if you are dealing with an unmanaged memory leak but that is much less likely unless your application does a lot of manual unmanaged memory management (e.g. object marshalling for P/Invoke).

Juncaceous

Already used SOS extension but all examples i tried does not helps me to isolate leak. – Savadove

As i said in my post i used almost all .NET managed memory profiles and they give shows me that my process has allocated about 10% of all memory process consume (according to task manager, perf.monitor counter, etc...) So i think it is really an unmanaged memory leak. – Savadove

run !dumpheap -stat and post the results. Don't just go on a hunch. Get hard data. If !dumpheap -stat and !eeheap -gc show that your managed heaps are small, then you can confidently move on to the unmanaged side. The allocators you cited above as the "leakers" are the functions via which the CLR allocates GC heap segments. I'd be surprised if this problem weren't a managed mem problem. – Filemon

Answer 3

using weak references may reduce memory usage http://en.wikipedia.org/wiki/Weak_reference .. that's very general, but once it saved me almost few GB s

There

I dont understand quite well how this helps me now? I have completed project (consist of 50+ .NET DLLs) and have memory leak somewhere, so i need to isolate it somehow.. – Savadove

by that i'm saying... that it is highly possible, that the leak could be caused by GC not collecting some obj references #707921 – There