external fragmentation and virtual address fragmentation in windbg

Question

I am using windbg to debug a memory issues on Win7.

I use !heap -s and got following output.

0:002> !heap -s
LFH Key                   : 0x6573276f
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
000f0000 00000002    1024    552   1024    257     7     1    0      0   LFH
00010000 00008000      64      4     64      2     1     1    0      0      
00330000 00001002    1088    160   1088      5     2     2    0      0   LFH
00460000 00001002     256      4    256      2     1     1    0      0      
012c0000 00001002    1088    408   1088      8    10     2    0      0   LFH
00440000 00001002    1088    188   1088     24     9     2    0      0   LFH
01990000 00001002    1088    188   1088     24     9     2    0      0   LFH
00420000 00001002    1088    152   1088      5     2     2    0      0   LFH
01d20000 00001002      64     12     64      3     2     1    0      0      
01c80000 00001002      64     12     64      1     2     1    0      0      
012e0000 00001002  776448 118128 776448 109939   746   532    0      0   LFH
    External fragmentation  93 % (746 free blocks)
    Virtual address fragmentation  84 % (532 uncommited ranges)
01900000 00001002     256      4    256      1     1     1    0      0      
01fa0000 00001002     256    108    256     58     3     1    0      0      
01c40000 00001002      64     16     64      4     1     1    0      0      
03140000 00001002      64     12     64      3     2     1    0      0      
33f40000 00001002      64      4     64      2     1     1    0      0      
340f0000 00001002    1088    164   1088      3     5     2    0      0   LFH
-----------------------------------------------------------------------------

My question is what is External fragmentation and what is Virtual addess fragmentation? And what does 93% and 84% mean?

Thank you in advance.

Skewer

Is there anyone here familiar with this? – Skewer

Answer 1

The output of WinDbg refers to the heap before the fragmentation numbers, in your case the heap 012e0000.

External fragmentation = 1 - (larget free block / total free size)

This means that the largest free block in that heap is 7.63 MB, although the total free size is 109 MB. This typically means that you can't allocate more than 7.63 MB in that heap at once.

For a detailed description of external fragmentation, see also Wikipedia.

Virtual address fragmentation: 1 - (commit size / virtual size)

While I have not found a good explanation for virtual memory fragmentation, this is an interpretation of the formula: virtual size is the total available memory. Commit size is what's used. The difference (1 - x) is unusable.

You can go into more details on that heap using !heap -f -stat -h <heap> (!heap -f -stat -h 012e0000 in your case).

Lindsy

Could you explain how do you know largest free block is 7.63MB? And what does Virtual size mean? In my case Virt is always equal to Reserv. – Skewer

@KevinTian: The largest free block can be calculated by solving the formula for largest free block. Largest free block = (1 - external fragmentation) * total free size. – Lindsy

@KevinTian: One distinguishes between physical memory, which can be used by Windows (kernel mode) and virtual memory, which can be used by Applications (user mode). As long as you don't do driver development, memory is the short term for virtual memory. WinDbg also displays the same values for Virt and Reserv on my machine. Maybe it's a bug of WinDbg 6.2, I also had different values before. – Lindsy

Answer 2

If you are trying to debug a memory fragmentation problem you should take a look at VMMAP from sysinternals.

http://technet.microsoft.com/en-us/sysinternals/dd535533

Not only you can see there the exact size of the largest free block, but you can also go to "Fragmentation view" in it to see visual presentation of how fragmented your memory is.

Showing

Answer 3

Thank Stas Sh 's answer.

I am using VMMap to analyze memory used by a process.

But I am confuse with Private Data displayed in VMMap.

I write a demo app, and use HeapCreate to create a private heap, and then allocate a lot of small blocks from that heap by HeapAlloc.

I use VMMap to analyze this demo app, and follow information is from VMMap.

Process: HeapOS.exe
PID: 2320

Type         Size        Committed  Private   Total WS  Private WS  Shareable WS  Shared WS  Locked WS  Blocks  Largest   
Total        928,388     806,452    779,360   782,544   779,144     3,400         2,720                 188     
Heap         1,600       500        488       460       452         8             8                     13      1,024
Private Data 888,224     774,016    774,016   774,016   774,012     4             4                     24      294,912

I found Heap is very small, but Private Data is very large.

But from Help of VMMap, it explained that

Private

Private memory is memory allocated by VirtualAlloc and not suballocated either by the Heap Manager or the .NET run time. 
It cannot be shared with other processes, is charged against the system commit limit, and typically contains application data. 

So I guess that Private Data is memory allocate by VirtualAlloc from virtual address space of process, and just can't be shared with other process. Private Data may be allocate by code of app, or by Heap Manager of OS or by .NET runtime.

Skewer