StringBuilder Class OutOfMemoryException

Question

I have written following function

public void TestSB()
{
  string str = "The quick brown fox jumps over the lazy dog.";
  StringBuilder sb = new StringBuilder();
  int j = 0;
  int len = 0;

  try
  {
     for (int i = 0; i < (10000000 * 2); i++)
     {
        j = i;
        len = sb.Length;
        sb.Append(str);
     }

    Console.WriteLine("Success ::" + sb.Length.ToString());
  }
  catch (Exception ex)
  {
      Console.WriteLine(
          ex.Message + " :: " + j.ToString() + " :: " + len.ToString());
  }
}

Now I suppose, that StringBuilder has the capacity to take over 2 billion character (2,147,483,647 to be precise).

But when I ran the above function it gave System.OutOfMemoryException just on reaching the capacity of about 800 million. Moreover, I am seeing widely different result on different PC having same memory and similar amount of load.

Can anyone please provide or explain me the reason for this?

Hiltonhilum

I would take a look at #364180 and #1734167 – Karlsbad

You will be able to approach the maimum better with StringBuilder sb = new StringBuilder(10000000 * 1); Using a(n initial) capacity is always a good idea with big collections. – Megargee

Answer 1

Each character requires 2 bytes (as a char in .NET is a UTF-16 code unit). So by the time you've reached 800 million characters, that's 1.6GB of contiguous memory required¹. Now when the StringBuilder needs to resize itself, it has to create another array of the new size (which I believe tries to double the capacity) - which means trying to allocate a 3.2GB array.

I believe that the CLR (even on 64-bit systems) can't allocate a single object of more than 2GB in size. (That certainly used to be the case.) My guess is that your StringBuilder is trying to double in size, and blowing that limit. You may be able to get a little higher by constructing the StringBuilder with a specific capacity - a capacity of around a billion may be feasible.

In the normal course of things this isn't a problem, of course - even strings requiring hundreds of megs are rare.

---

¹ I believe the implementation of StringBuilder actually changed in .NET 4 to use fragments in some situations - but I don't know the details. So it may not always need contiguous memory while still in builder form... but it would if you ever called ToString.

Noma

Whell, but why this behaviour can vary between different machines considering that allocation limit is handled by VM and not system itself? – Harmful

@Tigran: It can vary based on two things: the VM implementation (different major versions, different variations based on CPU architecture) and the implementation details of StringBuilder itself. Oh, and how much memory is available of course... – Noma

@Harmful I think because the StringBuilder effectively needs contiguous memory to allocate its contents and memory can be fragmented in different ways based on what the machine has been doing beforehand. You can get still possibly get OutOfMemory exceptions when there may be still be lots of physical RAM still free, because there is not enough contiguous memory. – Uria

@jon agree, but what sounds strange to me that the guy saying to have very different results on machines with apparently same config. But you confirm actually my doubts on machines equality. – Harmful

I thought the whole idea of the string builder is it created a collection of strings, and only joined them on the .ToString() operation, which is why it's so much faster than string concatenation? – Lightyear

@Rob: No - at least that's not how it used to work. The reason it's usually been faster than string concatenation is that it can use a single mutable buffer (grown only when required) - whereas with string concatenation, each operation requires all the data to be copied as strings are immutable. See pobox.com/~skeet/csharp/stringbuilder.html for more information. – Noma

I noticed one more thing on my machine. While running it with VS2008 (.Net 3.5) it is throwing exception at lesser value of loop(i) then with VS2010(.Net 4.0) – Hiltonhilum

@atur: Right - that corresponds with my footnote - the implementation of StringBuilder has changed in .NET 4. – Noma

The original example I tried was in .Net 4.0 where it was able to take upto 800 million. .Net 3.5 is crashing for 1/3 the value – Hiltonhilum