How to read a large (1 GB) txt file in .NET?

Question

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

private void ReadTxtFile()
{            
    string filePath = string.Empty;
    filePath = openFileDialog1.FileName;
    if (string.IsNullOrEmpty(filePath))
    {
        using (StreamReader sr = new StreamReader(filePath))
        {
            String line;
            while ((line = sr.ReadLine()) != null)
            {
                FormatData(line);                        
            }
        }
    }
}

In FormatData() I check the starting word of line which must be matched with a word and based on that increment an integer variable.

void FormatData(string line)
{
    if (line.StartWith(word))
    {
        globalIntVariable++;
    }
}

Stepp

You may want to post FormatData (or a simplified version), just in case. – Billiot

@Matthew: just ignore FormatData(), actually whole process is slow, so for troubleshooting i have commented it. – Stepp

You can't ignore the FormatData if you want a fast solution, you be best formatting the data in a separate thread from that reading the data. – Dipteran

You've not given much context of how you're accessing globalIntVariable. Given the implementation of FormatData, is it important that the lines are indeed read in order? If not reading multiple larger chunks of data and concurrently aggregating the global variable will be more efficient. – Dipteran

You should post actual benchmark data for solutions you have already tried. – Billiot

Answer 1

If you are using .NET 4.0, try MemoryMappedFile which is a designed class for this scenario.

You can use StreamReader.ReadLine otherwise.

Prent

If you're only doing sequential reading you're better of using StreamReader than MemoryMappedFile since it's much faster. Memory mapping is better for random access. – Meng

Furthermore you probably can't create a ViewAccesor spanning the entire 1 gb so you have to manage that as well as parsing out the linebreaks. FileStreams are 10 times as fast as Memory-Mapped files for sequential reading. – Meng

@ konrad - agreed, great comment, FYI there is a bit of a discussion of this in O'Reilly's excellent "C# 4.0 in a Nutshell", page 569. For sequential I/O and a 1GB file size, then MemoryMappedFiles are definitely overkill and may slow things down. – Peale

@TimSchmelter do you really expect to load 1 gb file to memory ?memorymappedfile has a lot of usages... i dont think this is one of them... – Frolick

@Peale i have tis book also , it doesnt say nothing about 1GB file. - the sample there is 1 million which is MB. the only thing mentiones is sequential vs random access. – Frolick

@RoyiNamir: MemoryMappedFile allow to read views of parts of extremly large files. You don't need to create a view from the whole file at once. So it's very scalable since you can define the portions yourself(f.e. 100MB).msdn.microsoft.com/en-us/library/dd997372.aspx – Gramnegative

@RoyiNamir whether the book (C# 4.0 in a Nutshell) has an example of exactly 1GB in size is irrelevant. There's actually a title on page 569 called "Memory Mapped Files and Random File I/O" I'm looking at it now. Quoted: "Rule of thumb: FileStreams are 10 times faster than MemoryMappedFiles for sequential I/O. MemoryMappedFiles are 10 times faster than FileStreams for random I/O". TL;DR Use the right tool for the right job. – Peale

@dodgy_coder: I'd be cautitous with such generalizations. Although i agree with your last sentence, but you should better measure it yourself. – Gramnegative

any full source code sample -using in real application in production environment, not msdn samples - about it ? – Manganin

Answer 2

Using StreamReader is probably the way to since you don't want the whole file in memory at once. MemoryMappedFile is more for random access than sequential reading (it's ten times as fast for sequential reading and memory mapping is ten times as fast for random access).

You might also try creating your streamreader from a filestream with FileOptions set to SequentialScan (see FileOptions Enumeration), but I doubt it will make much of a difference.

There are however ways to make your example more effective, since you do your formatting in the same loop as reading. You're wasting clockcycles, so if you want even more performance, it would be better with a multithreaded asynchronous solution where one thread reads data and another formats it as it becomes available. Checkout BlockingColletion that might fit your needs:

Blocking Collection and the Producer-Consumer Problem

If you want the fastest possible performance, in my experience the only way is to read in as large a chunk of binary data sequentially and deserialize it into text in parallel, but the code starts to get complicated at that point.

Meng

+1 The limiting factor is going to be the speed of the reads from disk, so to improve performance have different threads reading vs processing the lines. – Dipteran

@Meng the last part you wrote is actually what StreamReader does internally, so why bother? – Bussey

Answer 3

You can use LINQ:

int result = File.ReadLines(filePath).Count(line => line.StartsWith(word));

File.ReadLines returns an IEnumerable<String> that lazily reads each line from the file without loading the whole file into memory.

Enumerable.Count counts the lines that start with the word.

If you are calling this from an UI thread, use a BackgroundWorker.

Wing

Answer 4

Probably to read it line by line.

You should rather not try to force it into memory by reading to end and then processing.

Unwholesome

Answer 5

StreamReader.ReadLine should work fine. Let the framework choose the buffering, unless you know by profiling you can do better.

Billiot

StreamReader.ReadLine is fine for small file but when i tried it for large file then it is very slow some time not responding. – Stepp

@Mathew: posted code look at it, lines length are not fix some time line contain only 200 word and some time it will be 2000 or greater that it. – Stepp

2000 isn't a huge amount. That's only 20 KB, if we're talking English words. However, you still may want to call the FileStream constructor manually, specifying the buffer size. I also think FormatData may actually be the issue. That method doesn't keep all the data in memory, does it? – Billiot

@Matthew: i have commented FormatData() and it is still slow, no much significant difference with and without FormatData(). – Stepp

@Jeevan can you define "slow"? If you read [small file] in n time, then big file will be read in n * [big file]/[small file]. Maybe you are experiencing what's expected? – Gumwood

Answer 6

TextReader.ReadLine()

Cobweb

Answer 7

I was facing same problem in our production server at Agenty where we see large files (sometimes 10-25 gb (\t) tab delimited txt files). And after lots of testing and research I found the best way to read large files in small chunks with for/foreach loop and setting offset and limit logic with File.ReadLines().

int TotalRows = File.ReadLines(Path).Count(); // Count the number of rows in file with lazy load
int Limit = 100000; // 100000 rows per batch
for (int Offset = 0; Offset < TotalRows; Offset += Limit)
{
  var table = Path.FileToTable(heading: true, delimiter: '\t', offset : Offset, limit: Limit);

 // Do all your processing here and with limit and offset and save to drive in append mode
 // The append mode will write the output in same file for each processed batch.

  table.TableToFile(@"C:\output.txt");
}

See the complete code in my Github library : https://github.com/Agenty/FileReader/

Full Disclosure - I work for Agenty, the company who owned this library and website

Danell

Answer 8

My file is over 13 GB:

You can use my class:

public static void Read(int length)
    {
        StringBuilder resultAsString = new StringBuilder();

        using (MemoryMappedFile memoryMappedFile = MemoryMappedFile.CreateFromFile(@"D:\_Profession\Projects\Parto\HotelDataManagement\_Document\Expedia_Rapid.jsonl\Expedia_Rapi.json"))
        using (MemoryMappedViewStream memoryMappedViewStream = memoryMappedFile.CreateViewStream(0, length))
        {
            for (int i = 0; i < length; i++)
            {
                //Reads a byte from a stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream.
                int result = memoryMappedViewStream.ReadByte();

                if (result == -1)
                {
                    break;
                }

                char letter = (char)result;

                resultAsString.Append(letter);
            }
        }
    }

This code will read text of file from start to the length that you pass to the method Read(int length) and fill the resultAsString variable.

It will return the bellow text:

Ipecac

"It will return the bellow text:" What text? – Lordinwaiting

Answer 9

I'd read the file 10,000 bytes at a time. Then I'd analyse those 10,000 bytes and chop them into lines and feed them to the FormatData function.

Bonus points for splitting the reading and line analysation on multiple threads.

I'd definitely use a StringBuilder to collect all strings and might build a string buffer to keep about 100 strings in memory all the time.

Tombouctou