C# Out of memory exception - warning strategy

Question

Inside a complex multithreaded application I am hitting an out-of-memory exceptions maybe once a week. The application is sending/reading massive amounts of data via several sockets, where the read data gets cached to avoid network card buffer overruns.
What is the best strategy to analyse the memory exceptions? During normal run-time, the App is shown with a size of "Total bytes in all Heaps" of up to 1.5 Gigabyte in the Process explorer.
Could it be a strategy to have a thread which is polling either

GC.GetTotalMemory()

or

PrivateMemorySize64()

once a second to know when to start analyzing things? I have not looked into commercial profilers yet and I am a bit concerned of the performance impact of them which could give also wrong results regarding the actual problem analysis.

Terresaterrestrial

How does your caching work? I'd look into a different allocation strategy for your caching. – Newt

To clarify, do you know what causes the out of memory exception and are you looking just for a way to configure the caching in your application so that this doesn't happen? Or is it that you don't know what exactly causes it and you want tot debug the problem? – Heighho

Answer 1

Your memory is probably being fragmented from numerous string operations or other operations that create and release small blocks of memory, such as boxing/unboxing.

You will get this exception when the CLR cannot allocate a large enough free block of memory.

I use the "CLR Profiler" and check the memory allocations. If you see numerous white-spots (free blocks) and no large free blocks then you need to start looking at how you are allocating objects.

For instance, before assigning one string to another, check to see if the strings are different first. Using StringBuilder is all cases, eliminate boxing, and other memory optimizations.

I use this technique and completely eliminated the exceptions, except for a known issue with binary de-serialization.

Rediscover the Lost Art of Memory Optimization in Your Managed Code at http://msdn.microsoft.com/en-us/magazine/cc163856.aspx

Investigating Memory Issues at http://msdn.microsoft.com/en-us/magazine/cc163528.aspx

Performance Optimization in Visual Basic .NET at http://msdn.microsoft.com/en-us/library/aa289513(v=vs.71).aspx

Plication

Most strings should be relocatable, and thus shouldn't cause memory fragmentation. LoH fragmentation can be a problem, but if you're using many strings large enough to end on the LoH you are probably doing something wrong. – Newt

What do you mean by other memory optimizations? At one part I am using heavily splits, regex, stacks and lists. Need to recheck for boxing – Terresaterrestrial

You cannot relocate memory if there is not a block large enough. That is why the out-of-memory exception is thrown. I never seen LOH causing out-of-memory exceptions. That is what it is for, a heap were large blocks of memory can be allocated without the need to find a large free continuous block of memory. – Plication

@weismat; using jagged arrays instead of multi-dimentional arrays, avoiding creating objects repeatedly/needlessly, and on and on. See the links I added to the answer. Make sure you check out Investigating Memory Issues at <msdn.microsoft.com/en-us/magazine/cc163528.aspx> – Plication

@AMissico, I think you're not clear on the distinction between normal heap and large object heap (LOH). Objects in normal heap can be always relocated. Objects in LOH are never relocated. And when allocating from LOH, you need to find free continuous block of virtual memory, or you get OOM. That's why fragmentation of LOH can be a problem, but not fragmentation of normal heap. – Heighho

@Heighho Object in the normal heap cannot be relocated while they are fixed. This can happen with the fixed statement, certain gc handles, and in native interop. But it should usually be possible to relocate them. – Newt

@CodeInChaos, right, I forgot about those situations, but they should be quite rare. I admit I tend to overuse strong words like “always”. – Heighho

@svick: If objects in the heap can always be relocated, then out-of-memory exceptions would not be thrown when there is plenty of free memory availabe to the application domain. The LOH can grow getting additional memory from the heap. If the LOH cannot grow due to fragmentation in the heap, the out-of-memory exception occurs – Plication

Answer 2

You might consider installing the debugging tools for windows and using adplus

ADPlus.vbs (ADPlus) is a tool from Microsoft Product Support Services (PSS) that can troubleshoot any process or application that stops responding (hangs) or fails (crashes).

Basically, you can set that watching the application, and when it crashes, it will capture a dump, that you can then analyse using WinDBG/SOS.

Eveland

I might go this route - I guess I will add it into my schedulded task list, so that it will start daily as every day will require a fresh start (as the .exe starts daily as well). – Terresaterrestrial

Answer 3

You can use a MemoryFailPoint to try and give you some guarantees on a given operation

But I recommend isolating the process that your app is running in, moving to 64 bit if that's an option and you may need to look at reducing some performance to give you reasonable guarantee's of your memory usage.

Faqir

Answer 4

How about using weak references for caching? clicky

Southworth

Just FYI, this is from the Weak References Guildlines: "Avoid using weak references as an automatic solution to memory management problems. Instead, develop an effective caching policy for handling your application's objects." – Sanborn

There are not so many places where weak references might help - I think I would split the process into two first to double the available data space. I have avoided this for administrative reasons so far. – Terresaterrestrial