.NET application memory usage - high unused .NET and unmanaged memory and fragmentation

Question

I am using ANTS memory profiler to diagnose an increase in memory leak I am facing in one of my .NET 2.0 applications. I took 7 snapshots of the process over a period of 7.5 hours, and here is a tabular representation of the data obtained -

G1 reprsents generation 1 size and G2 generation 2 size. Except for the unmanaged space and private bytes, all other values are in MB.

My questions are -

1. Why is there such high unused .NET space even when the heap sizes are low ?

2. My large object heap goes to a maximum of some 2 MB, and during the last 3 snapshots remains at 96 KB. Then why is there such high large fragments, and are they responsible for the high unused space ?

3. The unmanaged space increases constantly. Is that responsible for an increase in private bytes over time ?

I am at my wit's end to solve this issue, and have performed several analyses but cant find a proper solution to this. I am ready to provide any other data needed.

Psychiatrist

That might be the result of objects staying in the memory. Maybe you should verify that all objects used is well disposed, as constant re-instanciation will imply more memory spaces being allocated. Which can be the cause of fragmentation also – Broccoli

There are very few objects on the LOH. The profiler shows an array that stays on it since the start of the application and another array is allocated and de-allocated at regular intervals which is expected. – Psychiatrist

Does the second array disposed correctly? GC in the LOH is very rare event. The possible scenario is that your second array gets allocated before the GC, so the possible scenario could be looked like: [1,2](the start of the app) [1,2,2](the 2nd array gets reallocation) When you get GC in the LOH, your LOH'd be looked like: [1,0,0,0,0,2] then you start allocate the second array once more, and get: [1,2,0,0,0,2] If the 2nd array hasn't constant length the things could be even worse. The idea is don't allow much reallocation in the LOH. – Philipson

@Philipson : I am calling dispose on the second array and all of its children. But yes, it can be of variable size. – Psychiatrist

@Cygnus: First of all I'd like to recommend to review this article: simple-talk.com/dotnet/.net-framework/… It gives you the picture what happens, when the reallocated array has the variable size. The second advice is do not use the arrays (in this case) they produce a lot of gaps in the LOH. Probably List, or BigList (from PowerCollections) could be the option. The point is do not call "new" each time your need the new collections. Use Clear() and write the data to the already allocated memory. Fught for the every byte you'd allocated! – Philipson

@Philipson : Thank you ! Ill definitely look into not using new everytime. What about the high unused space and increase in unmanaged space ? – Psychiatrist

One more thing. Lets say i create a List kept on the LOH and occupying 100kb, and then clear it at some point. Now, I dont use new and add members to the list so that the list size exceeds 100 kb. Wont that also lead to fragmentation ? – Psychiatrist

Yes expanding a List will lead to fragmentation. The point is to minimize fragmentation. On a List Clear does not release the capacity. You must call TrimExcess to release the memory. When you create the List make it an "optimal" capacity. – Bipod

@Psychiatrist 1) It's hard say for sure. As far as I know .net reserve some extra memory for each process, maybe it's the root of the problem of the unused space. 2) Yes this scenario could cause fragmentation in case if your list was internally reallocated. BigList use another reallocation mechanism and avoids this problem. – Philipson

Answer 1

As Alex already pointed out a very nice explanation of the problem class large object heap fragmentation is found here:

https://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/

The problem is well known in the .NET FX Dev Team and continuously been worked at. There is a good chance that the symptoms fade off using more recent FX releases.

Starting with .NET 4.5.1 there will be a GC method call to even compact the LOH: http://blogs.msdn.com/b/mariohewardt/archive/2013/06/26/no-more-memory-fragmentation-on-the-large-object-heap.aspx However, finding the root cause of the LOHF would be way more efficient than just wiping it of the heap wasting tons of ms's

Let me know, if you need further details how to isolate such effects.

Seb

Arleta