Memory leak debugging with windbg without user stack trace

Question

I have a full memory dump but in this instance I don't have a user stack trace database to go with it, I have up to date symbols and the original binaries that go with the dump, normally, I've been able to use the !heap -p -a address to view the call stack at the moment of allocation but this won't work without the user stack trace database.

My question is whether there's another way (albeit less direct approach) to get at the source of this memory leak.

        LFH Key                   : 0x0000005c2dc22701
Termination on corruption : ENABLED
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                            (k)     (k)    (k)     (k) length      blocks cont. heap 
-------------------------------------------------------------------------------------
00000000002e0000 00000002 3125248 3122792 3125248    282   378   197    0      7   LFH
0000000000010000 00008000      64      4     64      1     1     1    0      0      
0000000000530000 00001002    1088    416   1088     51    10     2    0      0   LFH
0000000000490000 00001002     512    284    512      5     5     1    0      0   LFH
0000000000af0000 00001002    1088    248   1088      2     2     2    0      0   LFH
0000000000c00000 00001002      64      8     64      3     1     1    0      0      
0000000000de0000 00001002     512      8    512      3     1     1    0      0      
0000000000ac0000 00001002   31616  30356  31616   1810    42     6    0      0   LFH
00000000012c0000 00001002     512      8    512      2     1     1    0      0      
0000000002140000 00001003     512     88    512     49     7     1    0    N/A   
0000000001ab0000 00001003     512      8    512      5     1     1    0    N/A   
00000000022f0000 00001003     512      8    512      5     1     1    0    N/A   
0000000002490000 00001003     512      8    512      5     1     1    0    N/A   
0000000000d40000 00001003     512      8    512      5     1     1    0    N/A   
0000000002690000 00001003     512      8    512      5     1     1    0    N/A   
0000000002860000 00001003     512      8    512      5     1     1    0    N/A   
0000000002e90000 00001002     512      8    512      2     2     1    0      0      
0000000002e10000 00001002    1536    556   1536     40     6     2    0      0   LFH
0000000001b90000 00011002     512      8    512      3     2     1    0      0      
00000000033e0000 00001002     512      8    512      3     2     1    0      0      
-------------------------------------------------------------------------------------

As you can see from this heap summary (!heap -s), heap 00000000002e0000 has grown pretty large, on closer inspection is can see that 70% of the data is allocated in blocks of size 0x4058, 0x23d1 and 0x10d1 (which is definitely some kind of pattern) so I'm pretty sure I want to investigate that further.

heap @ 00000000002e0000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    4058 1ea - 7b2870  (39.56)
    23d1 1dc - 42989c  (21.39)
    10d1 1ed - 20627d  (10.40)
    c51 1f4 - 180e34  (7.73)
    307 25b - 7217d  (2.29)
    378 1f9 - 6d7b8  (2.20)
    188 40e - 63570  (1.99)
    c0 59f - 43740  (1.35)
    30 12c7 - 38550  (1.13)
    28 147e - 333b0  (1.03)
    140 22a - 2b480  (0.87)
    138 231 - 2abb8  (0.86)
    2340 11 - 25740  (0.75)
    100 244 - 24400  (0.73)
    120 1ea - 22740  (0.69)
    78 456 - 20850  (0.65)
    1010 12 - 12120  (0.36)
    10188 1 - 10188  (0.32)
    10008 1 - 10008  (0.32)
    4000 4 - 10000  (0.32)

My problem is that I don't know where to go from here, previously I've followed the instructions found here with great success but this time around I don't have a user stack trace database and I can't easily reproduce this pattern but I know the memory dump contains a lot of useful information I'm just not sure how to go about getting at something meaningful from here. Windbg experts? Memory dump analysts? Please advice.

Some blocks, first couple of bytes

0:000> dc 0000000005254b80    
00000000`05254b80  52474d45 00000000 050f1c40 00000000  EMGR....@.......
00000000`05254b90  00000000 00000000 00000001 00000001  ................
00000000`05254ba0  00000400 000003ff 0001d4c0 00000001  ................
00000000`05254bb0  524d4954 00000000 051fcd10 00000000  TIMR............
00000000`05254bc0  f7b315d0 000007fe 05254b80 00000000  .........K%.....
00000000`05254bd0  00000000 00000000 05254bd8 00000000  .........K%.....
00000000`05254be0  05254bd8 00000000 05254be8 00000000  .K%......K%.....
00000000`05254bf0  05254be8 00000000 05254bf8 00000000  .K%......K%.....
0:000> dc 00000000051ce640    
00000000`051ce640  52474d45 00000000 04f1ab00 00000000  EMGR............
00000000`051ce650  00000000 00000000 00000001 00000001  ................
00000000`051ce660  00000400 000003ff 0001d4c0 00000001  ................
00000000`051ce670  524d4954 00000000 05037070 00000000  TIMR....pp......
00000000`051ce680  f7b315d0 000007fe 051ce640 00000000  ........@.......
00000000`051ce690  00000000 00000000 051ce698 00000000  ................
00000000`051ce6a0  051ce698 00000000 051ce6a8 00000000  ................
00000000`051ce6b0  051ce6a8 00000000 051ce6b8 00000000  ................
0:000> dc 0000000004fdb1f0    
00000000`04fdb1f0  52474d45 00000000 04f1b570 00000000  EMGR....p.......
00000000`04fdb200  00000000 00000000 00000001 00000001  ................
00000000`04fdb210  00000400 000003ff 0001d4c0 00000001  ................
00000000`04fdb220  524d4954 00000000 04ed6ba0 00000000  TIMR.....k......
00000000`04fdb230  f7b315d0 000007fe 04fdb1f0 00000000  ................
00000000`04fdb240  00000000 00000000 04fdb248 00000000  ........H.......
00000000`04fdb250  04fdb248 00000000 04fdb258 00000000  H.......X.......
00000000`04fdb260  04fdb258 00000000 04fdb268 00000000  X.......h.......
0:000> dc 0000000001e649b0    
00000000`01e649b0  52474d45 00000000 00351270 00000000  EMGR....p.5.....
00000000`01e649c0  00000000 00000000 00000001 00000001  ................
00000000`01e649d0  00000400 000003ff 0001d4c0 00000001  ................
00000000`01e649e0  524d4954 00000000 01e64130 00000000  TIMR....0A......
00000000`01e649f0  f7b315d0 000007fe 01e649b0 00000000  .........I......
00000000`01e64a00  00000000 00000000 01e64a08 00000000  .........J......
00000000`01e64a10  01e64a08 00000000 01e64a18 00000000  .J.......J......
00000000`01e64a20  01e64a18 00000000 01e64a28 00000000  .J......(J......

Petroleum

Debug Diag memory leak rule is a suitable approach in this case, iis.net/learn/troubleshoot/performance-issues/… – Whenever

Waiting this long to take a minidump only ensures you'll completely drown in the data. – Eleazar

@HansPassant Not sure if that's directed towards Lex or me but the circumstances surrounding the dump is not something I have the luxury of doing anything about, I'm just trying to determine how serious this issue is, if at all, serious. It could be an isolated incident for all that I know. – Petroleum

Answer 1

Use the !heap -flt s on the offending size(s) (with logging to file)

Then manually dump the contents on some of them and try to guess what kind of data they contain. If you are lucky it’s C++ objects with a vtable address in first DWORD which make them “easy” to recognize. If not, use dc , dds commands and try to figure out what the contents is.

Another approach is to find types which have corresponding size to those you suspect leaking.

============================Find symbols of spesific size===================================
0:011> dt -v -s a4 <MyDll>!*
Enumerating symbols matching <MyDll>!*, Size = 0xa4
Address   Size Symbol
           0a4 <MyDll>!NMDATETIMEFORMATW
           0a4 <MyDll>!CWinApp
           0a4 <MyDll>!CWinApp
==> Check all modules
!for_each_module ".echo @#ModuleName;dt -v -s a4  ${@#ModuleName}!*"

You can also try to find heap blocks which has a pointer to a leak suspect

0:008> !heap -srch 09C07058
_HEAP @ 02C90000
in HEAP_ENTRY: Size : Prev Flags - UserPtr UserSize - state
    0B7DA920: 002c : 002c [01] - 0B7DA928 (00000158) - (busy)
      diasymreader!Mod1::`vftable'

Mook

I tried dumping some of the larger blocks found some patterns, don't know the data, do you recognize any of it? – Petroleum

try: ln 00000000`52474d45 – Mook

Since your blocks is starting with the same 64 bits pointer, check this. The ln command will list nearest symbol(s). try: ln 00000000`05254b80 – Mook

Nothing happens :( I've triple checked that the symbols loaded successfully. – Petroleum

What about: !address 00000000`05254b80 It will show if it's a pointer to another heap block However, I think you have a difficult case here ! – Mook

It's a pointer to an address within the same heap. What's so interesting is that those blocks have really unique magic numbers in them but I have no idea where they originate from. EMGR and TIMR has to mean something to someone... also it cannot be a coincidence that they are aligned, 48 bytes a part exactly. – Petroleum

Hmm... I check the next to largest blocks and I'm seeing the following now, winhttp!HTTP_USER_REQUEST::`vftable', 0000000003e70530 0241 0241 [00] 0000000003e70540 023d1 - (busy), many of them as well, it might be related to my WinHttp client code... – Petroleum

I tracked down the issue. A couple of interfaces were missing virtual deconstructors, upon which these held on to winhttp handles indefinitely – Petroleum

How to determine what heap blocks are the biggest ones? From the OP's question the Reserved Size is about 3GB (!). But the sum of 'total' column of the !heap -stat output is only about 16 MB. Where're all other blocks that consume all the rest? – Gerundive