When should I ever use file.read() or file.readlines()?

Question

I noticed that if I iterate over a file that I opened, it is much faster to iterate over it without reading it first. That is:

l = open('file','r')
for line in l:
    ...

is much faster than

l = open('file','r')
for line in l.read(): 
    ...

or

l = open('file','r')
for line in l.readlines(): 
    ...

The 2nd loop will take around 1.5x as much time (I used timeit over the exact same file, and the results were 0.442 vs. 0.660), and would give the same result.

So, when should I ever use .read() or .readlines?

Pantelegraph

Please clarify. is the timeit measurement for read, or for readlines? I'd expect the read loop to take longer because it returns a single string, so iterating over it would go character-by-character. If your file has on average 100 characters per line, then the code in the for line in l.read() loop will execute a hundred times as many times as the code in the for line in l: loop. – Abaddon

it's also for readlines(). Surprisingly there's almost no time difference between read() or readlines()... – Pantelegraph

Answer 1

The short answer to your question is that each of these three methods of reading bits of a file have different use cases. As noted above, f.read() reads the file as an individual string, and so allows relatively easy file-wide manipulations, such as a file-wide regex search or substitution.

f.readline() reads a single line of the file, allowing the user to parse a single line without necessarily reading the entire file. Using f.readline() also allows easier application of logic in reading the file than a complete line by line iteration, such as when a file changes format partway through.

Using the syntax for line in f: allows the user to iterate over the file line by line as noted in the question.

(As noted in the other answer, this documentation is a very good read):

https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects

Note: It was previously claimed that f.readline() could be used to skip a line during a for loop iteration. However, this doesn't work in Python 2.7, and is perhaps a questionable practice, so this claim has been removed.

Clamorous

Mixing readline with a for loop over the file doesn't actually work; readline doesn't understand the next implementation's buffering. If you want to skip a line in a for loop, you should call next on the file. – Ogden

oh, also if you're using regular expressions to search over the string of the file, than it could be a bit faster than iterating over the file. – Pantelegraph

Answer 2

From the tutorial included in the Python documentation:

When size is omitted or negative, the entire contents of the file will be read and returned; it’s your problem if the file is twice as large as your machine’s memory.... For reading lines from a file, you can loop over the file object. This is memory efficient, fast, and leads to simple code:

>>> for line in f:
...     print(line, end='')
...
This is the first line of the file.
Second line of the file

Ethelinda

That's not an accurate description of the API for either C or Python. – Ogden

I figured I wouldn't explain it very well, that's why I pulled the rest of my answer straight from the documentation. – Ethelinda

C does not default to reading files line by line. There isn't even a standard function for reading files line by line at all in C; getline is a POSIX extension. Also, the loop over f.read() does not read the entire file on each iteration, and it does not iterate over the lines. – Ogden

I wasn't referring to getline, rather fscanf. – Ethelinda

It did last year when I took CS108, not sure when it changed, but I'll be sure to look into it. – Ethelinda

Let us continue this discussion in chat. – Ethelinda

Answer 3

#The difference between file.read(), file.readline(), file.readlines()
file = open('samplefile', 'r')
single_string = file.read()    #Reads all the elements of the file 
                               #into a single string(\n characters might be included)
line = file.readline()         #Reads the current line where the cursor as a string 
                               #is positioned and moves to the next line
list_strings = file.readlines()#Makes a list of strings

Milkfish

Answer 4

f.read():

Use Cases:

1. when reading the whole file as a string.
2. When you want to split the whole file string with a specified delimiter
3. When you want to use the Regular expression on the whole file to search for something.
4. There is only one I/O Operation.

f.readline()

Use Cases:

1. When you want to read one line at a time.
2. one I/O Operation at every line read.

f.readlines()

Use Cases:

1. return the list of lines in the file and include the new line character in each element of the list.
2. One I/O Operation.

Ann

for answering everything in the question, you may add for line in f which is basically iter(f). Use cases: 1. When you want to lazy iterate a file 2. One or Several I/O Operations (depending on implementation) – Keratinize

Answer 5

Note that readline() is not comparable to the case of reading all lines in for-loop since it reads line by line and there is an overhead which is pointed out by others already.

I ran timeit on two identical snippts but one with for-loop and the other with readlines(). You can see my snippet below:

  
def test_read_file_1():  
    f = open('ml/README.md', 'r')  
    for line in f.readlines():  
        print(line)  
  
  
def test_read_file_2():  
    f = open('ml/README.md', 'r')  
    for line in f:  
        print(line)  
  
  
def test_time_read_file():  
    from timeit import timeit  
  
    duration_1 = timeit(lambda: test_read_file_1(), number=1000000)  
    duration_2 = timeit(lambda: test_read_file_2(), number=1000000)  
  
    print('duration using readlines():', duration_1)  
    print('duration using for-loop:', duration_2)

And the results:

duration using readlines(): 78.826229238
duration using for-loop: 69.487692794

The bottomline, I would say, for-loop is faster but in case of possibility of both, I'd rather readlines().

Initiate

Answer 6

readlines() is better than for line in file when you know that the data you are interested starts from, for example, 2nd line. You can simply write readlines()[1:].

Such use cases are when you have a tab/comma separated value file and the first line is a header (and you don't want to use additional module for tsv or csv files).

Supertanker

I think you're getting readline() confused with readlines()? Also pretty sure this object isn't indexible post Python-3 – Shade

@JonathanSimon you are right, I changed readline -> readlines – Supertanker

It would be better to just skip the first line of the file explicitly, by reading it before the loop. The real reason for using .read or .readlines is because you actually want an in-memory copy for some other reason, as explained in the accepted answer. – Concentrate