Analyzing the root cause of OutOfMemoryException in WPF app with WinDbg

Question

I'm having some troubles to understand the crash dump and to find what is the root cause of the OutOfMemoryException thrown by the WPF application. The exception is thrown after the application has been run for several hours so this clearly indicates that there is a memory leak.

My first step was to look at !address -summary command :

--- Usage Summary ---------------- RgnCount ------- Total Size -------- %ofBusy %ofTotal
<unknown>                              2043          58997000 (   1.384 Gb)  71.43%   69.22%
Heap                                    152           fcc3000 ( 252.762 Mb)  12.74%   12.34%
Image                                  1050           bc77000 ( 188.465 Mb)   9.50%    9.20%
Stack                                   699           7d00000 ( 125.000 Mb)   6.30%    6.10%
Free                                    518           3f6b000 (  63.418 Mb)            3.10%
TEB                                     125             7d000 ( 500.000 kb)   0.02%    0.02%
Other                                    12             36000 ( 216.000 kb)   0.01%    0.01%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            2186          685b7000 (   1.631 Gb)  84.14%   81.53%
MEM_IMAGE                              1710           f3f3000 ( 243.949 Mb)  12.29%   11.91%
MEM_MAPPED                              186           46db000 (  70.855 Mb)   3.57%    3.46%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT                             3366          73fe7000 (   1.812 Gb)  93.52%   90.62%
MEM_RESERVE                             716           809e000 ( 128.617 Mb)   6.48%    6.28%
MEM_FREE                                518           3f6b000 (  63.418 Mb)            3.10%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1650          5e19e000 (   1.470 Gb)  75.87%   73.52%
PAGE_EXECUTE_READ                       224           bc42000 ( 188.258 Mb)   9.49%    9.19%
PAGE_READWRITE|PAGE_WRITECOMBINE         28           439f000 (  67.621 Mb)   3.41%    3.30%
PAGE_READONLY                           573           3d7b000 (  61.480 Mb)   3.10%    3.00%
PAGE_WRITECOPY                          214            f8f000 (  15.559 Mb)   0.78%    0.76%
PAGE_EXECUTE_READWRITE                  265            d0a000 (  13.039 Mb)   0.66%    0.64%
PAGE_READWRITE|PAGE_GUARD               357            33b000 (   3.230 Mb)   0.16%    0.16%
PAGE_EXECUTE_WRITECOPY                   55            119000 (   1.098 Mb)   0.06%    0.05%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown>                                   78d40000           2350000 (  35.313 Mb)
Heap                                        36db0000            fd0000 (  15.813 Mb)
Image                                       64a8c000            e92000 (  14.570 Mb)
Stack                                        4b90000             fd000 (1012.000 kb)
Free                                        7752f000            1a1000 (   1.629 Mb)
TEB                                         7ede3000              1000 (   4.000 kb)
Other                                       7efb0000             23000 ( 140.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

This shows that the memory is quite high.

Then I'm looking at GC heap size with eeheap -gc command. It shows that the heap is quite big (1.1GB) which indicates that there is a problem within the managed part of the application.

5fc90000  5fc91000  60c7acd4  0xfe9cd4(16686292)
5a060000  5a061000  5b05e9c0  0xffd9c0(16767424)
56de0000  56de1000  57ddf1c4  0xffe1c4(16769476)
57de0000  57de1000  58ddbbbc  0xffabbc(16755644)
73ff0000  73ff1000  74fe0f5c  0xfeff5c(16711516)
50de0000  50de1000  51dcfa58  0xfeea58(16706136)
5b060000  5b061000  5c05ca54  0xffba54(16759380)
4fde0000  4fde1000  50ddfd8c  0xffed8c(16772492)
Large object heap starts at 0x03921000
 segment     begin allocated  size
03920000  03921000  049013d0  0xfe03d0(16647120)
14850000  14851000  15837380  0xfe6380(16671616)
178d0000  178d1000  1889a3e0  0xfc93e0(16552928)
1a1c0000  1a1c1000  1b1abca8  0xfeaca8(16690344)
40de0000  40de1000  41dc8b48  0xfe7b48(16677704)
42de0000  42de1000  43827170  0xa46170(10772848)
54de0000  54de1000  55dd6d18  0xff5d18(16735512)
Total Size:              Size: 0x448fde94 (1150279316) bytes.
------------------------------
GC Heap Size:            Size: 0x448fde94 (1150279316) bytes.

Notice that there are 64 segments and each about (16MB). It seems that there is some data held in the memory and never released.

Next I look at !dumpheap -stat:

65c1f26c   207530     19092760 System.Windows.Media.GlyphRun
65c2c434   373991     20943496 System.Windows.Media.RenderData
68482bb0   746446     26872056 MS.Utility.ThreeItemList`1[[System.Double, mscorlib]]
65c285b4   746448     29857920 System.Windows.Media.DoubleCollection
64c25d58   299568     32353344 System.Windows.Data.BindingExpression
6708a1b8  2401099     38417584 System.WeakReference
67082c2c  1288315     41226080 System.EventHandler
67046f80  1729646     42238136 System.Object[]
64c1409c   206969     52156188 System.Windows.Controls.ContentPresenter
67094c9c   382163     64812664 System.Byte[]
004b0890      159     65181140      Free
64c150d0   207806     72316488 System.Windows.Controls.TextBlock
6708fd04  1498498     97863380 System.String
6848038c   847783    128775772 System.Windows.EffectiveValueEntry[]

As I understand it, there is no a signle object that takes all the memory. The biggest one is just about 122MB. Summing up all the sizes (8500 lines of outputed lines) gives the (1.1GB) of occupied memory. It seems that all the object graph is somehow duplicated and added to the memory and never released.

The !gcroot 6848038c or !gcroot 6708fd04 to inspect how EffectiveValueEntry and System.String are reachable, never ends, the stack is soooo big...

dumpheap -mt <address> doesn't show me something that srikes me. !finalizequeue shows that there are many objects (more that 2 millions) registered for finalization :

6708a1b8  2401099     38417584 System.WeakReference
Total 2417538 objects

I suspect that the OutOfMemoryException occures when the application tries to duplicate the object graph and allocate new memory, but I cannot find the root cause of it.

Question How can I drill down to the root of the problem (what other command of windbg can I use to check it). As it seems that not just one object is leaking but the whole object graph.. Am I on the right track or there is something else I'm overlooking ? What are other hypothesis ?

Broyles

Try using CLR Profiler as well, it gives you a lot of the complex information very easily. You can export the necessary logs for CLR Profiler by using !TraverseHeap. Most importantly, it makes it very easy to show the object allocation graphs. However, I'm affraid it might not help you - I think there was some annoying memory leak in WPF itself, looking around SO might help you find that question. – Jeffers

@Jeffers Unfortunatelly I have only the crash dump and can do the post-mortem debugging. Anyway thanks for the hit on CLR provider. Is the !TraverseHeap command specific to CLR Profiler ? – Megagamete

You can load the file produced by !TraverseHeap into CLR Profiler, so crash dump is quite enough - open it in WinDbg, run !TraverseHeap, and open the resulting file in CLR Profiler. You can also parse the file yourself, it's a simple text file. – Jeffers

But it's in an different module because just with SOS loaded I get in windbg "No export TraversHeap found" error – Megagamete

It might be in SOSex. Try googling, I'm sure you'll find what's wrong. Oh, and make sure you specify a filename as a parameter. – Jeffers

I've had a lot of help of this guide in the past: blogs.msdn.com/b/paullou/archive/2011/06/28/… – Motorboat

Answer 1

Object graph duplication

Your application uses ~1.1 GB of virtual memory by .NET. You can see that from the output of !eeheap -gc directly

GC Heap Size:            Size: 0x448fde94 (1150279316) bytes.

or by summing up the values of !dumpheap -stat.

Summing up all the sizes (8500 lines of output lines) gives the (1.1GB)

This roughly correlates to the value displayed as <unknown> in !address -summary.

--- Usage Summary ---------------- RgnCount ------- Total Size -------- %ofBusy %ofTotal
<unknown>                              2043          58997000 (   1.384 Gb)  71.43%   69.22%

There is no reason to assume that the whole object graph is being duplicated. This is a normal situation.

OutOfMemory

At the moment, there are 65 MB of virtual memory already committed by .NET and marked as free (from !dumpheap -stat):

004b0890      159     65181140      Free

Unfortunately, those 65 MB are split into 159 smaller regions. to get the largest block of those, you need to run a !dumpheap -mt 004b0890.

In addition, .NET could obtain another 63 MB from Windows (from !address -summary):

--- Usage Summary ---------------- RgnCount ------- Total Size --------
Free                                    518           3f6b000 (  63.418 Mb)

But the largest block is only 1.6 MB, so that's almost useless:

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                        7752f000            1a1000 (   1.629 Mb)

So, clearly, the application is out of memory.

Where is the memory?

252 MB are in native heaps. It seems you're using some native DLLs. While that does not seem too much at the moment, this fact might could indicate the presence of pinned objects. Pinned objects are not garbage collected. Look at the output of !gchandles to find out whether that could be part of the problem.

188 MB are in DLLs. You can unload native DLLs which are not in use, but for .NET assemblies, you probably can't do much about that.

125 MB are in stacks. With a default size of 1 MB, it seems that there are 125 threads in your application. Use !clrstack to find out what they are doing and why they did not finish yet. Potentially, each thread works on something and has not freed the objects yet. If you have it under your control, do not start so many threads in parallel. E.g. use 8 threads only and wait for threads to finish before doing the next piece of work.

Of course the majority of memory is used by .NET objects. However, you drew a few wrong conclusions.

As I understand it, there is no a single object that takes all the memory. The biggest one is just about 122MB.

Note that there is not a single object EffectiveValueEntry[] eating 122 MB ob memory. There are 847.783 of them. This changes the question from "Why does this object use so much memory?" to "Why are there so many of them?". For example, why does your application need 207.806 text blocks? Is it really displaying so much text?

Using !gcroot is a good idea. However, you used it with the address of a method table instead of an object:

!gcroot 6848038c
!gcroot 6708fd04

These were both numbers from the output of !dumpheap -stat. Using them in !gcroot should have given a warning like

Please note that 6848038c is not a valid object.

Instead, !gcroot works on individual objects only which you get from a !dumpheap without the -stat parameter.

You can use !traveseheap filename.log to dump all objects into a file compatible with CLR profiler [Codeplex]. Note that CLR Profiler cannot read the -xml format. After loading the object information, Heap Graph is probably the most useful button for you.

To find out the trigger of the OutOfMemoryException, you can use the !u command. You'll need to read some MSIL code to understand what it does. See also How to identify array type. However, in your scenario, I guess that's useless, because even small objects could trigger this.

Stockman

Hi @Thomas. Thank you for all this knowledge. I've made a mistake with gcroot command in my question. I'm doing before dumpheap -mt 6848038c then I pick the object address and gcroot it. But it never ends. I walks every thread – Megagamete

Answer 2

In addition to the other answer, you can try to visualize the memory adresses and the GC heap with the WinDbg extension cosos gcview

Eady