StringBuilder for string concatenation throws OutOfMemoryException

Question

We mostly tend to following the above best practice.

Have a look at String vs StringBuilder

But StringBuilder could throw OutOfMemoryException even when there is sufficient memory available. It throws OOM exception because it needs "continuous block of memory".

Some links for reference StringBuilder OutOfMemoryException

and there are many more.....

How many of you faced this problem or aware and what did you do to resolve it?

Is there anything I am missing?

P.S: I wasn't aware of this.

I have rephrased the question.

*** The same thing worked with manual concatenation(I'll verify this and update SO). The other thing that caused me concern was that there is enough memory in the system. That's the reason I raised this question here to check whether any one faced this problem or there was something drastically wrong with the code.

Schade

I had a similar issue, resolved it by specifying the StringBuilder's capacity up front, thus preventing the copying + allocations as the string grew. – Personage

Answer 1

The underyling string you create will also need a contiguous block of memory because it is represented as an array of chars (arrays require contiguous memory) . If the StringBuilder throws an OOM exception you woludn't be able to build the underlying without it.

If creating a string causes an OOM, there is likely a more serious issue in your application.

Edit in response to clarification:

There is a small subset of cases where building a string with a StringBuilder will fail when manual concatenation succeeds. Manual concatenation will use the exact length required in order to combine two strings while a StringBuilder has a different algorithmn for allocating memory. It's more aggressive and will likely allocate more memory than is actually needed for the string.

Using a StringBuilder will also result in a temporary doubling of the memory required since the string will be present in a System.String form and StringBuilder simultaneously for a short time.

But if one way is causing an OOM and the other is not, it still likely points to a more serious issue in your program.

Gleich

As far as I know, when StringBuilder.ToString() method is called, memory for a new string is not allocated, and data is not copied. String object that was created shares the same memory with StringBuilder, which enters a special state, "copy on change". So, memory will be doubled only if StringBuilder is modified after calling ToString(). – Tantalite

Not true, I checked StringBuilder source code and it allocates a new string and copies bytes using an unsafe pointer into the string memory. Maybe this was true back in 2009 but certainly not in 2020. – Newton

Answer 2

If StringBuilder is going to throw an OutOfMemoryException in your particular situation, then doing manual string concatenation is NOT a better solution; it's much worse. This is exactly the case (creating an extremely, extremely long string) where StringBuilder is supposed to be used. Manual concatenation of a string this large will take many times the memory that creation of a string with StringBuilder would take.

That said, on a modern computer, if your string is running the computer out of contiguous memory your design is deeply, deeply flawed. I can't imagine what you could possibly doing that would create a string that large.

Breadth

Answer 3

How much memory are we talking about? I'm not talking about free or total memory in the system, but how long is the string you're concatenating?

A memory overflow exception is almost always a very bad sign about your code, even if it fails long before the memory actually runs out, like you've experienced due to continous memory not being available.

At that point you should really restructure the code.

For instance, here are various ways to combat the problem:

1. Do not keep as much data in memory at one time, put it on disk or something
2. Break it up, keep a list of string/stringbuilders and only add to them up to a certain length before switching to a new one, keeps the "continous memory" problem at bay
3. Restructure the algorithm to not build up gigabyte of data in memory at all

Apical

Answer 4

If you are running soclose to your memory limits that this is even a concern, then you should probably think about a different architecture or get a beefier machine.

Peregrinate

A beefier machine will not help. OOM's are generated regardless of the underlying hardware. Memory is a virtual limitation. The only hardware part that would make a difference is 64 bit machine vs. 32 bit machine. – Gleich

Answer 5

If you look at how StringBuilder is implemented, you'll see that it actually uses a String to hold the data (String has internal methods, that will allow StringBuilder to modify in place).

I.e. they both use the same amount of memory. However, since StringBuilder will automatically extend the underlying array and copy as necessary when needed (but doubling the capacity) that is most likely the cause of the out of memory error. But as others have already pointed out both of the require a continuous block of memory,

Chalice

Answer 6

Well, the question actually is, why do you need to work with strings that long? If you stumble upon this problem, more than likely you should alter your concept.

This problems affects even the System.String class, so you should rather chunk your input into List< string> and process the data in parallel, which should increase overall performance if written properly.

Communal

Answer 7

I encountered this exception with very large strings built sucessively with different stringbuilders (which should not have caused a problem as they were declared within anonymous functions), and finally solved it by reusing a single StringBuilder, declared outside of the anonymous function.

Odalisque

Answer 8

I had a very similar experience where I was appending strings but forgot to add the String.Format. Thus:

myStringBuilder.Append("1,""{0}""", someVeryLargeIntVariable)

should have been:

myStringBuilder.Append(String.Format("1,""{0}""", someVeryLargeIntVariable))

Note that this is my vb.net code that failed. I replicated a similar test in c# with:

myStringBuilder.Append('a', 1564544656);

vs.

myStringBuilder.Append(string.Format("1,\"{0}\"", 1564544656));

But in my case, vb.net got me in trouble b/c of the implicit conversions (I couldn't parallel the exact same problem in c#).

I hope that helps someone.

Chevy

Using a 'String.Format' with a StringBuilder defeats the advantage of using the StringBuilder. – Isoclinal