Memory free up string builder an d byte[] in c#: Out of memory exception

Question

I am working on c#. I want know how i can free the stringbuilder n byte[]....Because I am getting an out of memory exception while using string builder.... Another thing is String.Replace() is also giving an out of memory exception or else is there any other way to do the same....please tell me how I can overcome these problems... thanks in advance

Sliding

Might be helpful if you clarify exactly what you are doing with StringBuilder when you run our of memory : for example, is this happening when you convert the StringBuilder into a string ? – Hitormiss

thanks for reply...Getting problem when doin stringbuilder.Append() also while doin StringBulder.Replace(oldStr,newStr); – Sliding

Answer 1

How big a string are you building? Do you know how big it will be up-front? If so set the Capacity of the StringBuilder.

If you don't do this it starts off with capacity of 16 and then doubles it each time you break through that limit. Capacity will go 16, 32, 64, 128.

Each time it does this it is likely to need to reallocate memory.

If the data is that big is it better to build it in a file stream rather than a StringBuilder?

Savannasavannah

Answer 2

If you are getting OOM, you are keeping too much data. It sounds to me like you should perhaps be using a TextWriter (to replace StringBuilder) or BinaryWriter / Stream (to replace byte[]) - in particular StreamWriter and FileStream (which write to a file).

For example:

using (TextWriter dest = File.CreateText("out.txt"))
{
    for (int i = 0; i < 100000; i++)
    {
        dest.Write(i);
        dest.Write(": ");
        dest.WriteLine(i % 5000); // why not
    }
}

The advantage of stream-based APIs like this is that once you've written a block of data it can be garbage collected etc.

Drawer

Answer 3

Here's some code to experiment with:

using System;
using System.Text;

class Program {
  static void Main(string[] args) {
    StringBuilder sb = new StringBuilder();
    try {
      //sb.Capacity = 690 * 1024 * 1024;
      while (true) sb.Append("asdf");
    }
    catch (OutOfMemoryException) {
      Console.WriteLine("Died at: {0:N0} characters", sb.Capacity);
      Console.WriteLine("Memory used: {0:N0} bytes", GC.GetTotalMemory(false));
      Console.ReadLine();
    }
  }
}

Output on my machine (Win7 32-bit):

Died at: 268,435,456 characters 
Memory used: 537,091,632 bytes

A 32-bit operating system provides a process with close to 2 gigabytes of virtual memory. As you can see, the sample program dies well short of this, consuming only 25% of it. The problem is virtual memory address space fragmentation. The available 2 gigabytes needs to store both code and data. You can get an insight into how the VM is carved up with SysInternal's VMMap utility.

There's something you can do about it with some insight in how StringBuilder works. It uses an internal array to store the string. This array is reallocated as needed to store the growing string, doubling the size each time. The Capacity property tells you the size of that array.

This causes another kind of fragmentation, heap fragmentation. After the array is doubled in size, the old array can be collected but produces a "free block" that can't be merged back to become available for a large contiguous allocation. That's an implementation detail of the Large Object Heap and the Windows heap manager, Google "look-aside cache" to find out more about it.

One workaround is to quickly gobble up virtual memory space before it can get fragmented. Remove the comment to see this at work. On my machine:

Died at: 723,517,440 characters
Memory used: 1,447,256,944 bytes

I fine-tuned the Capacity value by experimentation, you may have to do the same to make it work on your machine. The difference is pretty dramatic, almost 3 times as much. Beware that this is very unlikely to reproduce well in a real program, timing is critical. Or on another machine for that matter, be sure to stay well below the cut-off point. Realistically, you are risking OOM once the buffer you use gets past 134 million characters.

Abnaki

Answer 4

A StringBuilder object maximum capacity can be Int32.MaxValue - MaxCapacity: 2,147,483,647 - characters. If the StringBuilder object is instantiated by calling one of the StringBuilder class constructors which haven't MaxCapacity parameter, this object will use the default maximum capacity of Int32.MaxValue. If for StringBuilder object instantiation is used the constructor which defines MaxCapasity value, as for example

int capacity = 520;
int maxCapacity = 2048;
StringBuilder stringBuilder = new StringBuilder(capacity, maxCapacity);

and if the number of characters in this instantiated stringBuilder will exceed 2048, the StringBuilder object does not allocate additional memory but instead throws a StackOverflow exception. So if characters amount in instantiated StringBuilder object will exceed a default or set in a constructor maximum capacity, the error will be thrown. More about it: Instantiating a StringBuilder object

This problem can be solved by adding a code which, before characters adding into the StringBuilder object, checks do after this operation the maximum capacity will not be exceeded, and if it will be, the code can create a new StringBuilder object and add these characters into it. After all required characters will be added into one or more StringBuilder objects, this additional code will combine data from these StringBuilder objects into the required report.

Hysell