Login/Register
WinDbg Address Summary
Asked Answered
Solved .netdebuggingmemoryheap-memorywindbg
O

2

11

Our WCF service hosted in IIS crashes (w3wp.exe ~ 1.6 GB) as the user load increases. We have got a dump through Debug Diag and ran this command in WinDbg. This is the output:

0:000> !address -summary

                                     
Failed to map Heaps (error 80004005)

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unclassified>                         6007          57a86000 (   1.370 Gb)  85.37%   68.48%
Free                                    268          19519000 ( 405.098 Mb)           19.78%
Image                                   874           80da000 ( 128.852 Mb)   7.84%    6.29%
Stack                                  8022           64ce000 ( 100.805 Mb)   6.14%    4.92%
TEB                                    2674            a72000 (  10.445 Mb)   0.64%    0.51%
NlsTables                                 1             24000 ( 144.000 kb)   0.01%    0.01%
ActivationContextData                     4              b000 (  44.000 kb)   0.00%    0.00%
CsrSharedMemory                           1              7000 (  28.000 kb)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                           16360          5dd0f000 (   1.466 Gb)  91.37%   73.30%
MEM_IMAGE                              1188           8454000 ( 132.328 Mb)   8.05%    6.46%
MEM_MAPPED                               36            974000 (   9.453 Mb)   0.58%    0.46%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT                            14613          51e17000 (   1.279 Gb)  79.75%   63.97%
MEM_FREE                                268          19519000 ( 405.098 Mb)           19.78%
MEM_RESERVE                            2971          14cc0000 ( 332.750 Mb)  20.25%   16.25%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         8423          475ea000 (   1.115 Gb)  69.51%   55.76%
PAGE_EXECUTE_READ                       155           69f1000 ( 105.941 Mb)   6.45%    5.17%
PAGE_READWRITE|PAGE_GUARD              5319           1f24000 (  31.141 Mb)   1.90%    1.52%
PAGE_READONLY                           364           11bc000 (  17.734 Mb)   1.08%    0.87%
PAGE_WRITECOPY                          179            a16000 (  10.086 Mb)   0.61%    0.49%
PAGE_EXECUTE_READWRITE                  109            1fd000 (   1.988 Mb)   0.12%    0.10%
PAGE_EXECUTE_WRITECOPY                   64            149000 (   1.285 Mb)   0.08%    0.06%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unclassified>                               66f0000           c041000 ( 192.254 Mb)
Free                                        71c97000           4109000 (  65.035 Mb)
Image                                       51b0e000            da4000 (  13.641 Mb)
Stack                                         c90000             3d000 ( 244.000 kb)
TEB                                         7f370000              1000 (   4.000 kb)
NlsTables                                   7ffb0000             24000 ( 144.000 kb)
ActivationContextData                          70000              5000 (  20.000 kb)
CsrSharedMemory                             7f6f0000              7000 (  28.000 kb)
PEB                                         7ffd4000              1000 (   4.000 kb)

What I don't understand is unclassified. They are all huge! I ran !address and again got lot of unclassified. I am a beginner in WinDbg so don't know how to proceed from here with regards to memory leak.

!heap -s and !gchandles gives the following output and these numbers look very small.

0:000> !gchandles
GC Handle Statistics:
Strong Handles:       101224
Pinned Handles:       23
Async Pinned Handles: 2
Ref Count Handles:    4
Weak Long Handles:    0
Weak Short Handles:   0
Other Handles:        0
Statistics:
      MT    Count    TotalSize Class Name
67370460        1           12 System.Web.Hosting.AppManagerAppDomainFactory
79b5f8b4        1           16 System.Threading.RegisteredWaitHandle
7aa02d34        1           20 System.Net.TimerThread+TimerQueue
79b76668        1           20 System.Threading._ThreadPoolWaitOrTimerCallback
79b9a934        1           24 System.Threading.AutoResetEvent
79ba1b18        1           28 System.SharedStatics
6733fc00        3           36 System.Web.Hosting.ISAPIRuntime
79b9f5e8        5           60 System.Object
79b9fde8        1           84 System.ExecutionEngineException
79b9fd9c        1           84 System.StackOverflowException
79b9fd50        1           84 System.OutOfMemoryException
79b9fc0c        1           84 System.Exception
7988736c        2          136 System.Threading.OverlappedData
79b9fe34        2          168 System.Threading.ThreadAbortException
79ba3670       10          360 System.Security.PermissionSet
79ba0ec8        4          448 System.AppDomain
79b88df4       19         1520 System.Runtime.Remoting.ServerIdentity
6736eb50       54         1728 System.Web.NativeFileChangeNotification
67375ecc      489        95844 System.Web.HttpContext
79b9ffcc     2645       126960 System.Threading.Thread
79b56c28    10536       683588 System.Object[]
79baa808    87474      1749480 System.Threading._TimerCallback
Total 101253 objects
0:000> !heap -s
LFH Key                   : 0x866023d1
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
Virtual block: 020b0000 - 020b0000 (size 00000000)
Virtual block: 02250000 - 02250000 (size 00000000)
00080000 00000002   65536  42512  42512   1765   248     1    2  21d36   L  
00180000 00008000      64     12     12     10     1     1    0      0      
00260000 00001002    1088    628    628     24     0     1    0      0   L  
00500000 00000002    1024     20     20      2     1     1    0      0   L  
00640000 00001002     256     28     28      1     1     1    0      0   L  
00680000 00001002     256     12     12      4     1     1    0      0   L  
006c0000 00001002     256     12     12      4     1     1    0      0   L  
00700000 00001002    1280   1172   1172     42    10     1    0      0   L  
00740000 00001002     256     12     12      4     1     1    0      0   L  
00ad0000 00001002    1280    296    296      2     1     1    0      0   L  
00b10000 00001002     256     12     12      4     1     1    0      0   L  
00b50000 00001002     256     12     12      4     1     1    0      0   L  
00b90000 00001002     256     12     12      4     1     1    0      0   L  
00bd0000 00001002     256     12     12      4     1     1    0      0   L  
00c10000 00001002     256     12     12      4     1     1    0      0   L  
00c50000 00001002     256     12     12      4     1     1    0      0   L  
00dc0000 00001002     256     12     12      4     1     1    0      0   L  
00e00000 00001002     256     12     12      4     1     1    0      0   L  
00e40000 00001002     256     12     12      4     1     1    0      0   L  
00e80000 00001002     256     12     12      4     1     1    0      0   L  
00ec0000 00001002     256     12     12      4     1     1    0      0   L  
00f00000 00001002     256     12     12      4     1     1    0      0   L  
01050000 00001002    7424   4628   4628     14     4     1    0      0   L  
01090000 00001002     256     36     36      1     1     1    0      0   L  
01110000 00001002    7232   3192   3192     70    41     1    0      0   L  
01120000 00001002     256     12     12      4     1     1    0      0   L  
01160000 00001002     256    176    176      2     1     1    0      0   L  
011a0000 00001002     256     24     24      3     1     1    0      0   L  
012e0000 00001002     256     12     12      3     1     1    0      0   L  
01320000 00001002     256     12     12      4     1     1    0      0   L  
01360000 00001002     256     12     12      4     1     1    0      0   L  
013a0000 00001002     256     12     12      4     1     1    0      0   L  
013e0000 00001002     256     72     72      3     1     1    0      0   L  
01420000 00001002     256     12     12      4     1     1    0      0   L  
01460000 00001002     256    168    168      3     1     1    0      0   L  
014a0000 00001002     256     12     12      4     1     1    0      0   L  
014e0000 00001002     256     12     12      4     1     1    0      0   L  
01520000 00001002     256     12     12      4     1     1    0      0   L  
01560000 00001002     256     12     12      4     1     1    0      0   L  
01670000 00001002     256     12     12      4     1     1    0      0   L  
016b0000 00001002     256     12     12      4     1     1    0      0   L  
016f0000 00001002     256     12     12      4     1     1    0      0   L  
01730000 00001002     256     12     12      4     1     1    0      0   L  
01770000 00001002     256     12     12      4     1     1    0      0   L  
017b0000 00001002     256     12     12      4     1     1    0      0   L  
017f0000 00001002     256     12     12      4     1     1    0      0   L  
01830000 00011002     256     12     12      4     1     1    0      0   L  
01870000 00001002    1088    120    120      7     1     1    0      0   L  
01880000 00001002      64     12     12      4     1     1    0      0   L  
01890000 00001002     256     12     12      4     1     1    0      0   L  
019d0000 00001002     256     12     12      4     1     1    0      0   L  
01a10000 00001002     256    116    116      0     0     1    0      0   LFH
01a50000 00001002     256     12     12      4     1     1    0      0   L  
01b50000 00001002     256    200    200      4     0     1    0      0   L  
01bb0000 00001002    3136   1500   1500     69    41     1    0      0   L  
01bc0000 00041002     256     12     12      4     1     1    0      0   L  
01c00000 00001002    3136   1500   1500     69    41     1    0      0   L  
01c10000 00041002     256     12     12      4     1     1    0      0   L  
01c50000 00041002     256     16     16      1     1     1    0      0   L  
01e10000 00041002     256     64     64      1     1     1    0      0   L  
01ed0000 00001002    1088    424    424     20     1     1    0      0   L  
1af40000 00001002     256    152    152      0     0     1    0      0   L  
1b060000 00001002      64     48     48     40     2     1    0      0   L  
1bab0000 00001002    1280    688    688     28     2     2    0      0   L  
-----------------------------------------------------------------------------

!eeheap -gc gives the following output:

0:000> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (000e9678)
generation 0 starts at 0x3415e208
generation 1 starts at 0x340b6894
generation 2 starts at 0x026f0038
ephemeral segment allocation context: none
 segment     begin allocated  size
026f0000  026f0038  066eda74  0x3ffda3c(67099196)
336c0000  336c0038  3471a7c4  0x105a78c(17147788)
Large object heap starts at 0x126f0038
 segment     begin allocated  size
126f0000  126f0038  1272cd38  0x3cd00(249088)
Heap Size:               Size: 0x5094ec8 (84496072) bytes.
------------------------------
Heap 1 (000ea8b0)
generation 0 starts at 0x39fd9748
generation 1 starts at 0x39f13224
generation 2 starts at 0x066f0038
ephemeral segment allocation context: none
 segment     begin allocated  size
066f0000  066f0038  0a6ef668  0x3fff630(67106352)
38e00000  38e00038  3a63e8bc  0x183e884(25421956)
Large object heap starts at 0x146f0038
 segment     begin allocated  size
146f0000  146f0038  146f0048  0x10(16)
Heap Size:               Size: 0x583dec4 (92528324) bytes.
------------------------------
Heap 2 (000ebdb8)
generation 0 starts at 0x3087fedc
generation 1 starts at 0x307e8868
generation 2 starts at 0x0a6f0038
ephemeral segment allocation context: none
 segment     begin allocated  size
0a6f0000  0a6f0038  0e6ee9f4  0x3ffe9bc(67103164)
2f3b0000  2f3b0038  31acc084  0x271c04c(41009228)
Large object heap starts at 0x166f0038
 segment     begin allocated  size
166f0000  166f0038  166f0048  0x10(16)
Heap Size:               Size: 0x671aa18 (108112408) bytes.
------------------------------
Heap 3 (000ed4d0)
generation 0 starts at 0x6d1c892c
generation 1 starts at 0x6d110038
generation 2 starts at 0x0e6f0038
ephemeral segment allocation context: none
 segment     begin allocated  size
0e6f0000  0e6f0038  126ee814  0x3ffe7dc(67102684)
2b1a0000  2b1a0038  2d14311c  0x1fa30e4(33173732)
6d110000  6d110038  6d75f814  0x64f7dc(6617052)
Large object heap starts at 0x186f0038
 segment     begin allocated  size
186f0000  186f0038  186f0048  0x10(16)
Heap Size:               Size: 0x65f10ac (106893484) bytes.
------------------------------
GC Heap Size:            Size: 0x175de850 (392030288) bytes.
Overture answered 9/8, 2012 at 3:48 Comment(2)
What are you trying to achieve? Investigate the crash or investigate memory usage? What is the call stack at the crash? – Ferrosilicon
Actually just trying to find out why the memory usage of w3wp.exe IIS 6 keeps going up? – Overture
A
10

About unclassified, a lot of posts on the Internet show that in late versions of WinDBG unclassified entries has just replaced the things that were mapped to different regions before. In previous versions of debugger you had these RegionUsageIsVAD, RegionUsageImage.

On my side, I also have a lot or unclassified entries in !address -summary output, but it doesn't prevent me from future debugging.

No, go back to your other results:

According to my experience with WinDBG, if eeheap shows ~300Mb of memory when MEM_COMMIT gives 1.3Gb, it could be a native memory leak.

See CodeProject on how you could catch it.

Note, that you possibly won't have stacks when running !heap -p -a, so you need to run your process with proper gflags before debug. You can read more about it

Then, you may possibly got another, simpler situation with repeated strings. I've run into such situation one time and described it.

Aquiculture answered 9/8, 2012 at 14:16 Comment(0)
U
3

WCF (Windows Communication Foundation) is a .NET technology. .NET does not make use of the Windows Heap Manager, so using any !heap command will not work for .NET. No wonder that it's small.

Instead, the .NET framework has its own memory manager due to garbage collection. You already used the correct command !eeheap -gc to display information about the .NET heaps. The last line of that command says:

GC Heap Size:            Size: 0x175de850 (392030288) bytes.

So .NET is responsible for almost 400 MB of <unclassified> (<unknown>) memory. The rest could be caused by

  • direct calls to VirtualAlloc() of course
  • allocations via the Windows Heap Manager that are larger than 512 kb (see the statement by Sasha Goldshtein). This could be some C++ code, for example. Working with images or video could often fall into this category.
  • some versions of MSXML
  • potential other garbage collecting memory managers, e.g. the heap manager from Java (a guess, I never verified, but the Windows Heap Manager does not work well for garbage collecting)

If these values remain constant, there's nothing to worry about. But if it increases, this might indicate a leak. I personally consider those 87000 timer callbacks as a potential source. What are all those timer callbacks doing? Are they ever able to finish their work?

79baa808    87474      1749480 System.Threading._TimerCallback
Ungenerous answered 30/5, 2017 at 20:0 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.