How to Solve Gen2 Heap Fragmentation

Question

I am running a C# application that services HTTP requests. I have noticed recently that it's taking up more memory then I expect. I grabbed some dumps, popped them in Windbg, and found that most of the memory was marked as Free:

!dumpheap -stat
...
00007ffde4783630   681599     65433504 System.Threading.Tasks.TaskFactory+CompleteOnInvokePromise
00007ffde47cc988   167885     76872908 System.Byte[]
00007ffde47c6948   521353     80352802 System.String
0000007e3a16c2d0  1870425   1415374334      Free

So the dump is ~3GB so about half of it is free memory. Looking at the heaps I see this:

!heapstat
Heap             Gen0         Gen1         Gen2          LOH
Heap0        82248472      7354560    987275056    178834656
Heap1        93146552      6382864    857470096    129435960
Total       175395024     13737424   1844745152    308270616

Free space:                                                 Percentage
Heap0        40969256       146456    640426720     54829792 SOH: 63% LOH: 30%
Heap1        75943736        94448    550812312     54825216 SOH: 65% LOH: 42%
Total       116912992       240904   1191239032    109655008

So the my small object heaps are very fragmented, specifically Gen2. On the server I can see that gen2 collections are happening (using performance counters) but even though they are the it looks like the gen2 heap is not being compacted. Even when there is only 1-2% of RAM available on the server the gen2 heap is not being compacted.

To me it looks like I am suffering this memory pressure because the heap is fragmented. However I can't figure out why the fragmentation is happening or why gen2 is unable to be compacted. Some of the free spaces are 6MB in size so I would think it would for sure compact those spaces away.

Can anyone give me some ideas on how to figure out why my heap is so fragmented? Am I even barking up the right tree here?

Any help will be greatly appreciated, thanks!

EDIT 1:

The breakdown of !gchandles is:

Handles:
Strong Handles:       4507
Pinned Handles:       58
Async Pinned Handles: 977
Ref Count Handles:    1
Weak Long Handles:    6087
Weak Short Handles:   724

Periodicity

Please show us your code. – Parasitology

Do either of these help? msdn.microsoft.com/en-us/library/… blogs.msdn.microsoft.com/mariohewardt/2013/06/26/… – Parasitology

@Parasitology unfortunately its a large application and it's not practical to post the code. I'm more looking for general things I can do to try and figure the fragmentation out. I'll read through the two posts you sent and get back to you. – Periodicity

@Parasitology I'm not sure compacting the LOH would get me the most bang for my buck. Looking at the numbers in my post its less fragmented then the Gen2 SOH and it's overall smaller. I'll give it a shot and see though. – Periodicity

Fair call @Periodicity . I wish you the best in solving your issue. – Parasitology

@Parasitology I appreciate the replies and will try compacting the LOH to see if it helps. – Periodicity

That is a lot of async pinned handles. Your program probably has a lot of active socket reads, using the same byte[] buffer over and over again. So the buffer is constantly pinned and the heap segment in which it is allocated cannot be recycled. Google ".net http buffer pooling" to get ahead. – Davis

Answer 1

The next step would be to use !gchandles and look for pinned handles. This might identify objects that are locked to a specific memory position, because some native code (e.g. C++) needs to access it. While garbage colelction may move objects around in memory, the C++ pointer will not be updated by .NET, so pinning is the only option.

Even when there is only 1-2% of RAM available on the server the gen2 heap is not being compacted.

You're talking about physical RAM here. From an operating system, I expexct that it uses the available resources as good as possible, so I expect the RAM to be 100% used at all the time. If it is "not used", I expect the OS to use it for caching. Due to this, the physical RAM usage cannot really be an indicator for garbage collection.

Also, I would not worry too much. If the memory is not used by .NET, it will be swapped to disk by the OS and thus free physical RAM for other purposes.

Steinman

I'll run !gchandles and post the results back when I'm back at the office. I get your point about physical RAM and that unused RAM is wasted RAM but what I am seeing is frequent gen2 GCs that are causing pauses in the application. What I believe is happening is because the physical RAM is mostly used the GC is running the expensive gen2 GC more often to try and free some RAM. However when I look at it most of the gen2 heap is already free and not being used. Is that the wrong conclusion? – Periodicity

Added the output of !gchandles. I take it I'll need to look at both Pinned and Async Pinned handles? Do I need to be concerned about the other types? – Periodicity

@shortspider: for the moment, see the comment by Hans Passant. I'll try to edit my answer accordingly. – Steinman