How can I read, replace and write very large files? [duplicate]

Question

Possible Duplicate:
How to read a large (1 GB) txt file in .NET?

What is the optimal way in C# to read an file, replace some strings and write in another new file? I need to do this with very large files like 8GB or 25GB.

Scrambler

plain text, db backup for sanitize. – Scrambler

Answer 1

There isn't much you can optimize about the I/O, most of the optimization should be on the string comparison to determine if the string should be replaced or not, basically you should do this

protected void ReplaceFile(string FilePath, string NewFilePath)
{
   using (StreamReader vReader = new StreamReader(FilePath))
   {
      using (StreamWriter vWriter = new StreamWriter(NewFilePath))
      {
         int vLineNumber = 0;
         while (!vReader.EndOfStream)
         {
            string vLine = vReader.ReadLine();
            vWriter.WriteLine(ReplaceLine(vLine, vLineNumber++));
         }
      }
   }
}
protected string ReplaceLine(string Line, int LineNumber )
{
   //Do your string replacement and 
   //return either the original string or the modified one
   return Line;
}

What is your criteria to find and replace a string?

Footpound

If you don't need the line number, you should remove it from the ReplaceLine() declaration and from the calling to that function – Footpound

Great answer. Shouldn't cause an exception from reading too much into memory per line. – Yong

Answer 2

I did 200GB files before. Please use StreamReader and StreamWriter.

StreamReader

StreamWriter

Cate

Answer 3

Are there lines that are not overly large? If so, you can read in the file line by line, do the replace on that line, and then write out that line to the new file. Since it's streamed very little memory is required.

Giovanna