Login/Register
Why do the output of !address and !heap not match?
Asked Answered
memorywindbg
E

1

6

I am debugging the following C program in Windbg:

int main()
{
    size_t size = 500*1024*1024;
    void *p = malloc(size);
    memset(p, 'a', size);
    printf("%p", p);
}

I compiled the program using: cl /Zi leak.c and there is a leak.exe generated.

I set a breakpoint at the line printf. The I run the following command:

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                     21          5f0f6000 (   1.485 Gb)           74.27%
Heap                                      3          1f501000 ( 501.004 Mb)  95.07%   24.46%
<unknown>                                39           1436000 (  20.211 Mb)   3.84%    0.99%
Image                                    35            300000 (   3.000 Mb)   0.57%    0.15%
MappedFile                                4            182000 (   1.508 Mb)   0.29%    0.07%
Stack                                     3            100000 (   1.000 Mb)   0.19%    0.05%
Other                                     6             3f000 ( 252.000 kb)   0.05%    0.01%
TEB                                       1              1000 (   4.000 kb)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                              21          207f0000 ( 519.938 Mb)  98.66%   25.39%
MEM_IMAGE                                64            54c000 (   5.297 Mb)   1.01%    0.26%
MEM_MAPPED                                7            1be000 (   1.742 Mb)   0.33%    0.09%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                 21          5f0f6000 (   1.485 Gb)           74.27%
MEM_COMMIT                               73          1f9c9000 ( 505.785 Mb)  95.98%   24.70%
MEM_RESERVE                              19           1531000 (  21.191 Mb)   4.02%    1.03%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                           27          1f45f000 ( 500.371 Mb)  94.95%   24.43%
PAGE_EXECUTE_READ                         9            376000 (   3.461 Mb)   0.66%    0.17%
PAGE_READONLY                            26            1e4000 (   1.891 Mb)   0.36%    0.09%
PAGE_WRITECOPY                            9              c000 (  48.000 kb)   0.01%    0.00%
PAGE_READWRITE|PAGE_GUARD                 2              4000 (  16.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                        1fb51000          5590f000 (   1.337 Gb)
Heap                                          750000          1f401000 ( 500.004 Mb)
<unknown>                                   7f0e0000            f00000 (  15.000 Mb)
Image                                       77bc0000             d6000 ( 856.000 kb)
MappedFile                                  7efe5000             fb000 (1004.000 kb)
Stack                                         210000             fd000 (1012.000 kb)
Other                                       7efa0000             33000 ( 204.000 kb)
TEB                                         7efdd000              1000 (   4.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

And I can see the heap is about 500MB, this is as expected.

But the !heap command cannot see this info:

There is only 1 heap.

0:000> !heap
Index   Address  Name      Debugging options enabled
  1:   00650000                 tail checking free checking validate parameters

0:000> !heap -a 00650000
Index   Address  Name      Debugging options enabled
  1:   00650000 
    Segment at 00650000 to 00750000 (0000f000 bytes committed) // Why so few memory committed.
    Flags:                40000062
    ForceFlags:           40000060
    Granularity:          8 bytes
    Segment Reserve:      00100000
    Segment Commit:       00002000
    DeCommit Block Thres: 00000200
    DeCommit Total Thres: 00002000
    Total Free Size:      00000517
    Max. Allocation Size: 7ffdefff
    Lock Variable at:     00650138
    Next TagIndex:        0000
    Maximum TagIndex:     0000
    Tag Entries:          00000000
    PsuedoTag Entries:    00000000
    Virtual Alloc List:   006500a0
    Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 00750000
    Uncommitted ranges:   00650090
            0065f000: 000f1000  (987136 bytes)
    FreeList[ 00 ] at 006500c4: 0065b340 . 0065b340  
        0065b338: 00458 . 028b8 [104] - free

    Segment00 at 00650000:
        Flags:           00000000
        Base:            00650000
        First Entry:     00650588
        Last Entry:      00750000
        Total Pages:     00000100
        Total UnCommit:  000000f1
        Largest UnCommit:00000000
        UnCommitted Ranges: (1)

0:000> dt p
Local var @ 0x30ff04 Type void*
0x00750020 
Void
0:000> !heap -p -a 0x00750020 
    address 00750020 found in
    _HEAP @ 650000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        00750018 3e80200 0000  [00]   00750020    1f400000 - (busy VirtualAlloc)
0:000> !heap -s 
NtGlobalFlag enables following debugging aids for new heaps:
    tail checking
    free checking
    validate parameters
LFH Key                   : 0x343f9ad2
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
Virtual block: 00fa0000 - 00fa0000 (size 00000000)
006d0000 40000062    1024     36   1024      1     1     1    1      0      
-----------------------------------------------------------------------------

Why I cannot see the info above in !heap -s? How to dump all entries in heap?

Eructate answered 5/6, 2015 at 15:22 Comment(3)
Which version of WinDbg do you use? You say that you can't see above info in !heap -s but you didn't include the output of !heap -s. – Severally
Hello Thomas, I am using WinDbg 6.3.9600.17298x86. Output of "!heap -s" is added. And Thank you for editing this question for me. – Eructate
Thanks for the additional information. A !address 00fa0000 for the virtual block should reveal the 500 MB. – Severally
S
3

It needs a bit of explanation first ...

Heap vs. Virtual memory

IMHO the heap was invented because the granularity of VirtualAlloc() (which is 64kB) was too wasteful for small amounts of data. If I look at the output of a !heap <address>, I can see

Heap entries for Segment00 in Heap 00100000
     address: psize . size  flags   state (requested size)
    00100000: 00000 . 00588 [101] - busy (587)

See the size column which has 5 digits. This probably means that the maximum useful size of a heap entry is

0:001> ? fffff
Evaluate expression: 1048575 = 000fffff

roughly 1 MB. Above that size, it probably doesn't make sense to have the granularity of a heap and you can use VirtualAlloc() directly. Or, not you, but the malloc() function decides that for you.

An error message in the output

In your output, have you noticed the two lines

Virtual Alloc List:   006500a0
Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 00750000

This is an indicator that there are parts in a heap that are actually handled by VirtualAlloc(). However, WinDbg is unable to find the datatype for it:

0:001> dt nt!_HEAP_VIRTUAL_ALLOC_ENTRY
Symbol nt!_HEAP_VIRTUAL_ALLOC_ENTRY not found.

The output of !heap -s is missing, so I created it on my machine (WinDbg 6.3.9600.16384):

0:001> !heap 00100000 -s
LFH Key                   : 0x62502d13
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
Virtual block: 005b0000 - 005b0000 (size 00000000)
Virtual block: 006b0000 - 006b0000 (size 00000000)
...

Virtual block: 1fab0000 - 1fab0000 (size 00000000)
00100000 00000002    1024    204   1024      4     7     1  500      0   LFH
-----------------------------------------------------------------------------

In my demo, I created ~500 blocks of 1 MB. Note that the size of 00000000 seems broken. However, if I use !address on a block, it get's the region size right (00100000):

0:001> !address 005b0000                                     
...
Usage:                  Heap
Base Address:           005b0000
End Address:            006b0000
Region Size:            00100000
State:                  00001000    MEM_COMMIT
Protect:                00000004    PAGE_READWRITE
Type:                   00020000    MEM_PRIVATE
Allocation Base:        005b0000
Allocation Protect:     00000004    PAGE_READWRITE

How could the missing _HEAP_VIRTUAL_ALLOC_ENTRY look like?

0:001> dd 770000 L10
00770000  00870000 00670000 00000000 00000000
00770010  00100000 00100000 c8d50110 04000000
00770020  003df5a8 00870020 00000000 00000000
00770030  000ffc00 00000001 000000c1 fdfdfdfd

The value 870000 at offset 0 seems to be an FLink to the next memory block and the value 670000 at offset 4 a BLink to the previous one.

Offset 10 and 14 would both match to the region size.

0:001> ? 0n1023*0n1024
Evaluate expression: 1047552 = 000ffc00

At offset +30 is the size (I allocated blocks of 1023 kB here).

FDFDFDFD is a well known debugging magic number indicating no-mans-land, therefore this is likely the end of the structure.

Extracting the structure from an XP dump

I have a dump available from Windows XP. See how the nt!_HEAP_VIRTUAL_ALLOC_ENTRY resolves there:

0: kd> dt -r1 nt!_HEAP_VIRTUAL_ALLOC_ENTRY
   +0x000 Entry            : _LIST_ENTRY
      +0x000 Flink            : Ptr32 _LIST_ENTRY
      +0x004 Blink            : Ptr32 _LIST_ENTRY
   +0x008 ExtraStuff       : _HEAP_ENTRY_EXTRA
      +0x000 AllocatorBackTraceIndex : Uint2B
      +0x002 TagIndex         : Uint2B
      +0x004 Settable         : Uint4B
      +0x000 ZeroInit         : Uint8B
   +0x010 CommitSize       : Uint4B
   +0x014 ReserveSize      : Uint4B
   +0x018 BusyBlock        : _HEAP_ENTRY
      +0x000 Size             : Uint2B
      +0x002 PreviousSize     : Uint2B
      +0x000 SubSegmentCode   : Ptr32 Void
      +0x004 SmallTagIndex    : UChar
      +0x005 Flags            : UChar
      +0x006 UnusedBytes      : UChar
      +0x007 SegmentIndex     : UChar

Conclusion

!heap seems to have problems for large blocks created by malloc() which actually uses VirtualAlloc() in this case. WinDbg cannot find the datatype which it expects for mapping the memory content to. Microsoft should probably fix this bug.

The output statistics for heaps of !address -summary seems to be reasonable.

Severally answered 5/6, 2015 at 21:28 Comment(1)
Thanks, Thomas. I can learn a lot from your answer. But I am still curious about !heap and !address. Let's wait if someone know why. – Eructate

© 2022 - 2024 — McMap. All rights reserved.