Large Object Heap and String Objects coming from a queue

Question

I have a windows console app that is supposed to run without restarts for days and months. The app retrieves "work" from an MSMQ and process it. There are 30 threads that process a work chunk simultaneously.

Each work chunk coming from the MSMQ is approximately 200kb most of which is allocated in a single String object.

I have noticed that after processing about 3-4 thousands of these work chunks the memory consumption of the application is ridiculously high consuming 1 - 1.5 gb of memory.

I run the app through a profiler and noticed that most of this memory (maybe a gig or so) is unused in the large object heap but the structure is fragmented.

I have found that 90% of these unused (garbage collected) bytes were previously allocated String. I started suspecting then that the strings coming in from the MSMQ were allocated, used and then deallocated and are therefore the cause of the fragmentation.

I understand that things like GC.Collect(2 or GC.Max...) wont help since they gc the large object heap but don't compact it (which is the problem here). So I think that what I need is to cache these Strings and re-use them somehow but since Strings are immutable I would have to use StringBuilders.

My question is: Is there anyway to not change the underlying structure (i.e. using the MSMQ as this is something I cant change) and still avoid initializing a new String everytime to avoid fragmenting the LOH?

Thanks, Yannis

UPDATE: About how these "work" chunks are currently retrieved

Currently these are stored as WorkChunk objects in the MSMQ. Each of these objects contains a String called Contents and another String called Headers. These are actual textual data. I can change the storage structure to something else if needed and potentially the underlying storage mechanism if needed to something else than an MSMQ.

On the worker nodes side currently we do

WorkChunk chunk = _Queue.Receive();

So there is little we can cache at this stage. If we changed the structure(s) somehow then I suppose we could do a bit of progress. In any case, we will have to sort out this problem so we will do whatever is needed to avoid throwing out months of work.

UPDATE: I went on to try some of the suggestions below and noticed that this issue cannot be reproduced on my local machine (running Windows 7 x64 and 64bit app). this makes things so much more difficult - if anyone knows why then it would really help repdocung this issue locally.

Abutter

How do you receive those strings? Once they are strings you're stuck. I they come from a stream or byte[] you may have some options. – Juggle

Hi Henk - Have a look at the update to get more info about these work chunks – Abutter

But is it an actual problem? 1.5GB on a 64bit PC with >= 8GB RAM should be able to continue. – Juggle

but eventually will slow to a crawl due to excessive paging...this is not a philosophical issue - it happens daily! – Abutter

Can you occasionally stop accepting jobs, finish all the jobs you are running (deallocating all your objects), run GC.Collect() and then start accepting jobs again? – Goosander

Yeah - you reckon that may work? there are several DoWork threads running on several such apps. so rejecting work on one of these apps temporarily is not an issue. – Abutter

The actual problem statement still isn't part of the question. Also what Windows version (ie Server 2008?) is the PC running, what Framework version (3.5/4). They all have subtle differences in GC. – Juggle

Its Windows Server 2008 and Framework 4.0 - I noticed a huge difference running an (accurate) simulation in my local machine running Windows 7 64 bit – Abutter

Answer 1

Your problem appears to be due to memory allocation on the large object heap - the large object heap is not compacted and so can be a source of fragmentation. There is a good article here that goes into more detail including some debugging steps that you can follow to confirm that fragmentation of the large object heap is happening:

Large Object Heap Uncovered

You appear to have two three solutions:

1. Alter your application to perform processing on chunks / shorter strings, where each chunk is smaller than 85,000 bytes - this avoids the allocation of large objects.
2. Alter your application to allocate a few large chunks of memory up-front and re-use those chunks by copying new messages into the allocated memory instead. See Heap fragmentation when using byte arrays.
3. Leave things as they are - As long as you don't experience out of memory exceptions and the application isn't interfering with other applications running on the system you should probably leave things as they are.

Its important here to understand the distinction between virtual memory and physical memory - even though the process is using a large amount of virtual memory, if the number of objects allocated is relatively low then it cam be that the physical memory use of that process is low (the un-used memory is paged to disk) meaning little impact on other processes on the system. You may also find that the "VM Hoarding" option helps - read "Large Object Heap Uncovered" article for more information.

Either change involves changing your application to perform either some or all of its processing using byte arrays and short substrings instead of a single large string - how difficult this is going to be for you will depend on what sort of processing it is that you are doing.

Barbarism

Thanks Justin. The problem is that these Strings come from a different system through a message queue. So I cant say "get half of that Work chunk" currently unless I change the overall storage structure - and i guess thats where i need ideas and suggestions – Abutter

@Abutter If you want to alter your application then it does look that way - for suggestions on how you might want to do this a bit more detail on the sort of processing that is being done is probably needed. Have you seen my latest edit? You should consider that this behaviour you are seeing may be perfectly fine (as long as you don't get OOM exceptions, is this a 32 bit or a 64 bit process?) – Barbarism

Justin - This is a 64bit process and the result is that the computer (Windows 2008 Server) slows to a crawl due to excessive paging. This makes sense. Let me ask this: if I change the String Contents property to char[][] that contains char arrays of char chunks of 85k (the limit to put something on the LOH) - would that help? – Abutter

@Abutter Yeah thats the sort of thing I'm talking about, except you need to be careful that the CLR doesn't just treat a multi-dimensional array as a single allocation, a list of arrays may be better. Also be aware that sizeof(char) == 2. – Barbarism

I know that List is using an array behind the scenes. Maybe a LinkedList<char[]> would be better in this case. Will give it a try over the weekend but its really a pain to test these kind of stuff – Abutter

Answer 2

When there is fragmentation on the LOH, it means that there are allocated objects on it. If you can affort the delay, you can once in a while wait till all currently running tasks are finished and call GC.Collect(). When there are no referenced large objects, they will all be collected, effectively removing the fragmentation of the LOH. Of course this only works if (allmost) all large objects are unreferenced.

Also, moving to a 64 bit OS might also help, since out of memory due to fragmentation is much less likely to be a problem on 64 bits systems, because the virtual space is almost unlimited.

Rid

Steven I think you are wrong since fragmentation doesnt mean the objects are there (in the LOH) but they were once there and eventually got de-allocated thus leaving a an empty chunk in the LOH. This means that if there is a chunk of 120k (say) and we are trying to allocate 121k then this will be allocated at the first available contiguous chunk of 121k bytes thus leaving the 120k chunk empty. GC.Collect() will unfortunately only de-allocate LOH objects (and for that GC.Collect(GC.MaxGeneration) is needed) and not compact the LOH. – Abutter

I don't think Steven is saying that GC.Collect will compact, I think he is saying call it when you only have a few objects on the go. That way it will get rid of the big objects between which are the gaps leaving you with a nice(ish) clean slate. – Goosander

@Yannis: What I'm saying is: an empty LOH can not be fragmented. Joey rephrased it nicely. – Rid

got it now - apologies for misunderstanding this initially. will give it a go as well – Abutter

@Yannis: Miscommunication is always caused by the sender. No apologies needed. – Rid

Answer 3

Perhaps you could create a pool of string objects which you can use whilst processing the work then return back once you've finished.

Once a large object has been created in the LOH, it can't be removed (AFAIK), so if you can't avoid creating these objects then the best plan is to reuse them.

If you can change the protocol at both ends, then reducing your 'Contents' string into a set of smaller ones (<80k each) should stop them from being stored in the LOH.

Airily

That's what the OP already said. But how do you reuse a string? – Juggle

Tony - the problem is serializing these contents and deserializing them at the other end. Whatever I do this object will contain these "contents" one way or another - even in small chunks. – Abutter

Can you process the data without having it hold it in a single string? i.e. can you walk the small chunks and 'consume' them without needing to concatenate first? – Airily

yeah - i m thinking of creating chunks of char[][] i.e. as many char array chunks required to store the string contents that are less than the 85k limit to put objects on the LOH. do you think that would work? – Abutter

We implemented something similar but with List<> of List<>s when reaching the threshold and it did avoid putting the object onto the LOH, so yeah, I think chunking into char[] should work – Airily

Answer 4

How about using String.Intern(...) to eliminate duplicates references. It has a performance penalty, but depending on your strings it might have an impact.

Sapphirine

It would work better if you could chop up your header and contents into key/value pairs, doing .Intern on all keys and values. You would then end up with no duplicate data, but a different data structure, that might require more processing. – Sapphirine