StringBuilder.ToString() throws OutOfMemoryException

Question

I have a created a StringBuilder of length "132370292", when I try to get the string using the ToString() method it throws OutOfMemoryException.

StringBuilder SB = new StringBuilder();

for(int i =0; i<=5000; i++)
{
    SB.Append("Some Junk Data for testing. My Actual Data is created from different sources by Appending to the String Builder.");
}

try
{
    string str = SB.ToString(); // Throws OOM mostly
    Console.WriteLine("String Created Successfully");
}
catch(OutOfMemoryException ex)
{
    StreamWriter sw = new StreamWriter(@"c:\memo.txt", true);
    sw.Write(SB.ToString()); //Always writes to the file without any error
    Console.WriteLine("Written to File Successfully");
}

What is the reason for the OOM while creating a new string and why it doesn't throw OOM while writing to a file?

Machine Details: 64-bit, Windows-7, 2GB RAM, .NET version 2.0

Suppository

Whats the content of mytext? And why are you writing to a StringBuilder if and then to a stream? Why no the stream directly using a StringWriter? – Oregano

Why that exaggerated length? You just reserved 126 MB of memory. – Parvati

Can you post the actual code from your program? – Sutter

My Actual code is very different than this, I'm creating the StringBuilder from different sources by Appending the values, and the final Length of the SB is "132370282". SB.ToString() works fine some times and fails some times. – Suppository

That's 252MB; why would you want a 252MB string? This seems an insanely bad idea - you should really be writing to the writer (sw) cumulatively - not building the entire thing in memory. (/cc @DebugErr just to note that this is 252MB, not 126MB) – Announcer

As an unrelated question: why would you install a 64-bit OS on a system with 2GB of memory? – Announcer

I was using 4GB on 64-bit machine. Since I was not able to reproduce the error, I wanted to test this scenario on extreme conditions(that's 2GB VM), and found this. – Suppository

possible duplicate of C# Stringbuilder System.OutOfMemoryException – Colecolectomy

Answer 1

What is the reason for the OOM while creating a new string

Because you're running out of memory - or at least, the CLR can't allocate an object with the size you've requested. It's really that simple. If you want to avoid the errors, don't try to create strings that don't fit into memory. Note that even if you have a lot of memory, and even if you're running a 64-bit CLR, there are limits to the size of objects that can be created.

and why it doesn't throw OOM while writing to a file ?

Because you have more disk space than memory.

I'm pretty sure the code isn't exactly as you're describing though. This line would fail to compile:

sw.write(SB.ToString());

... because the method is Write rather than write. And if you're actually calling SB.ToString(), then that's just as likely to fail as str = SB.ToString().

It seems more likely that you're actually writing to the file in a streaming fashion, e.g.

using (var writer = File.CreateText(...))
{
    for (int i = 0; i < 5000; i++)
    {
        writer.Write(mytext);
    }
}

That way you never need to have huge amounts of text in memory - it just writes it to disk as it goes, possibly with some buffering, but not enough to cause memory issues.

Verso

If he calls ToString in both cases, wouldn't the fact that he's writing to disk be an irrelevant detail here? – Sutter

Both probably off - 64bit process would have plenty of memory, but likely runs as 32bit and hits address space fragmentation; it likely succeeds to write to disk because it already converted result to string once and previous call did not fail (based on sample code) – Georgiegeorgina

@AlexeiLevenkov: The sample code isn't the real code anyway, but even in a 64-bit CLR, object size is limited. Also, if you call ToString() on a StringBuilder twice, it still creates two strings - at least in the tests I've just run... (Given the use of .NET 2.0, that might not have been the case back then...) – Verso

@JonSkeet Agreed, didn't spot the lowercase w, and since this means it's obviously not the correct code, all bets are off. – Sutter

@JonSkeet I believe that it may be because of the GC, i.e, when the code is executed in the try block(in the actual app) the availability of free(contiguous) memory may cause OOM and the virtual memory reaches a critical situation and this invokes the GC implicitly before the catch block, GC may free certain amount of memory which is sufficient when the code runs in catch block. – Suppository

@Thiru: Oh, there's a catch block suddenly, is there? It's a good idea to indicate when you've changed a question significantly after answers have been posted. The GC should collect before the OOM is thrown anyway. I've seen rare occasions when it doesn't, but it's been a long time since I've seen that. Can you reproduce this in a short but complete program? The answer is still the same though, basically - don't try to create such huge strings. – Verso

Answer 2

Workaround: Suppose you would want to write a big string stored in StringBuilder to a StreamWriter, I would do a write this way to avoid SB.ToString's OOM exception. But if your OOM exception is due to StringBuilder's content add itself, you should work on that.

public const int CHUNK_STRING_LENGTH = 30000;
while (SB.Length > CHUNK_STRING_LENGTH )
{
    sw.Write(SB.ToString(0, CHUNK_STRING_LENGTH ));
    SB.Remove(0, CHUNK_STRING_LENGTH );
}
sw.Write(SB);

Cazzie

This was a helpful solution for me in getting to the final solution. I just want to point out 3 errors. First, it should be (SB.Length > 0) as the condition otherwise you will lose the last few thousand pieces of data. Second, you may be on your final run, which means you can't take the full chunk length, and instead you should check for this case and use the remaining length (to avoid an OutOfArgumentRangeException). Third, I believe the last line is surplus and an accident. – Roseanneroseate

You probably have to think the way it is coded, rather than the way you want the code to be. I believe my above code works well and encourage you to debug and verify it. – Cazzie

Yes you are right, it is correct. And it looks better to do it your way. I'm sorry I doubted you :). Although I would name the variables better (stringBuilder, make a "buffer" variable for the ToString). – Roseanneroseate

Answer 3

You have to remember that strings in .NET are stored in memory in 16-bit unicode. This means string of length 132370292 will reqire 260MB of RAM.

Furthermore, while executing

string str = SB.ToString();

you are creating a COPY of your string (another 260MB).

Keep in mind that each process have its own RAM limit so OutOfMemoryException can be thrown even if you have some free RAM left.

Blackamoor

each char is 2 bytes, so you need to double all those numbers – Announcer

Answer 4

Might help someone , if your logic needs large objects then you can change your application to 64bit and also
change your app.config by adding this section

  <runtime>  
    <gcAllowVeryLargeObjects enabled="true" />  
  </runtime> 

gcAllowVeryLargeObjects On 64-bit platforms, enables arrays that are greater than 2 gigabytes (GB) in total size.

Confirmed

Answer 5

String m_filename = "c:\temp\myfile.xml"
StreamWriter sw   = new StreamWriter(m_filename);
while (sb.Length  > 0)
{
     int writelen = Math.Min(sb.Length, 30000);
     sw.Write       (sb.ToString(0, writelen));
     sb.Remove      (0,writelen);
}
sw.Flush();
sw.Close();
sw = null;

Counterproof

btw this code block is in "CarlsonNotation" tm. Unlike other coding styles it's much easier to read because of the way the '=' and { and function calls are alligned. This is because when you really start scanning a lot of code as fast as you can it's easier for the eye to follow. – Counterproof