Generator expressions vs. list comprehensions

Question

When should you use generator expressions and when should you use list comprehensions in Python?

# Generator expression
(x*2 for x in range(256))

# List comprehension
[x*2 for x in range(256)]

Gamma

could [exp for x in iter] just be sugar for list((exp for x in iter)) ? or is there an execution difference ? – Argo

it think I had a relevant question, so when using yield can we use just the generator expression from a function or we have to use yield for a function to return generator object ? – Floris

@Argo Very late answer to your comment: in Python2 there is a tiny difference, the loop variable will leak out of a list comprehension, while a generator expression will not leak. Compare X = [x**2 for x in range(5)]; print x with Y = list(y**2 for y in range(5)); print y, the second will give an error. In Python3, a list comprehension is indeed the syntactic sugar for a generator expression fed to list() as you expected, so the loop variable will no longer leak out. – Vanden

I'd suggest reading PEP 0289. Summed up by "This PEP introduces generator expressions as a high performance, memory efficient generalization of list comprehensions and generators". It also has useful examples of when to use them. – Gilbye

@Gilbye I'm also eight years late to the party, and the PEP link was perfect. Thanks for making that easy to find! – Interminable

What about list(exp for x in iter)? I'm guessing, it's alternative syntax for list((exp for x in iter))? It's somewhat confusing though, as brackets are part of the syntax for generator expressions, as well as the usual function application syntax. – Barbet

@NaderGhanbari: When a generator expression is the only argument to a callable (like list), the call parentheses can serve double duty as the genexpr parentheses (specifically, the genexpr requires parentheses on either side, and no comma on either side inside the parens). They don't own their parentheses, they just need them to exist. So both options are legal and identical, the former being more idiomatic. – Vivien

Answer 1

John's answer is good (that list comprehensions are better when you want to iterate over something multiple times). However, it's also worth noting that you should use a list if you want to use any of the list methods. For example, the following code won't work:

def gen():
    return (something for something in get_some_stuff())

print gen()[:2]     # generators don't support indexing or slicing
print [5,6] + gen() # generators can't be added to lists

Basically, use a generator expression if all you're doing is iterating once. If you want to store and use the generated results, then you're probably better off with a list comprehension.

Since performance is the most common reason to choose one over the other, my advice is to not worry about it and just pick one; if you find that your program is running too slowly, then and only then should you go back and worry about tuning your code.

Griffith

Sometimes you have to use generators -- for example, if you're writing coroutines with cooperative scheduling using yield. But if you're doing that, you're probably not asking this question ;) – Jase

I know this is old, but I think it's worth noting that generators (and any iterable) can be added to lists with extend: a = [1, 2, 3] b = [4, 5, 6] a.extend(b) -- a will now be [1, 2, 3, 4, 5, 6]. (Can you add newlines in comments??) – Karoline

@Karoline your example belies the words you are saying. There's also a fine point here. Lists can be extended with generators, but then there was no point in making it a generator. Generators cannot be extended with lists, and generators are not quite iterables. a = (x for x in range(0,10)), b = [1,2,3] for instance. a.extend(b) throws an exception. b.extend(a) will evaluate all of a, in which case there's no point in making it a generator in the first place. – Hagar

@SlaterTyranus you're 100% correct, and I upvoted you for accuracy. nevertheless, I think his comment is a useful non-answer to the OP's question because it will help those who find themselves here because they typed something like 'combine generator with list comprehension' into a search engine. – Needlework

Wouldn't the reason for using a generator to iterate through once (e.g. my concern about lack of memory overrides my concern about "fetching" values one at a time) probably still apply when iterating multiple times? I'd say it might make a list more useful, but whether that's enough to outweigh memory concerns is something else. – Weariless

The reason to use a generator expression instead of a list comprehension is less performance but rather memory consumption in case of huge iterations. – Proponent

@SlaterVictoroff generators can be extended with lists (by creating a new generator): itertools.chain(range(10), [4, 5, 6]) – Stevenson

I also like generators for this idiom to get the fisrt matching item from a list of objects : next(x for x in some_list if x.foo == "bar") since it's neater than the comprehension equivalent [x for x in some_list if x.foo == "bar"][0] because it's easier to supply a default for no match like: next((for x in some_list if x.foo == "bar"), None). For a working example try print(next((x for x in range(10) if x == 11), 100)) and see how it defaults to 100 since 11 is not in the list. – Pelham

Answer 2

Iterating over the generator expression or the list comprehension will do the same thing. However, the list comprehension will create the entire list in memory first while the generator expression will create the items on the fly, so you are able to use it for very large (and also infinite!) sequences.

Elliotelliott

+1 for infinite. You can't do that with a list, regardless of how little you care about performance. – Mister

Can you create infinite generators using the comprehension method? – Garthgartner

@Annan Only if you already have access to another infinite generator. For example, itertools.count(n) is an infinite sequence of integers, starting from n, so (2 ** item for item in itertools.count(n)) would be an infinite sequence of the powers of 2 starting at 2 ** n. – Ephebe

A generator deletes items from memory after their iterated over. So its fast if you have big data you just want to display it, for example. Its not a memory hog. with generators items are processed 'as needed'. if you want to hang on to the list or iterate over it again (so store the items) then use list comprehension. – Gallstone

Answer 3

Use list comprehensions when the result needs to be iterated over multiple times, or where speed is paramount. Use generator expressions where the range is large or infinite.

See Generator expressions and list comprehensions for more info.

Leucocyte

This will be probably a little off-topic, but unfortunately "un-googlable"... What would "paramount" mean in this context? I'm not a native english speaker... :) – Medley

@GuillermoAres this is direct result of "googling" for meaning of paramount: more important than anything else; supreme. – Mcknight

So lists are speedier than generator expressions? From reading dF's answer, it came across that it was the other way around. – Goodoh

It's probably better to say that list comprehensions are faster when the range is small, but as the scale increases it becomes more valuable to compute the values on the fly -- just in time for their use. That's what a generator expression does. – Leveille

OK, but what is better to prefer (as a default) when it fits none of the two categories? – Stevenson

The "speed is paramount" confused me too. Other recent answers confirm (with %timeit benchmarks) what this implies - that list comprehension is slightly faster than a generator (as @HassanBaig said, this is not what I expected). But list comprehension memory consumption (with %memit) is much higher than a generator. – Luminescent

Answer 4

The important point is that the list comprehension creates a new list. The generator creates a an iterable object that will "filter" the source material on-the-fly as you consume the bits.

Imagine you have a 2TB log file called "hugefile.txt", and you want the content and length for all the lines that start with the word "ENTRY".

So you try starting out by writing a list comprehension:

logfile = open("hugefile.txt","r")
entry_lines = [(line,len(line)) for line in logfile if line.startswith("ENTRY")]

This slurps up the whole file, processes each line, and stores the matching lines in your array. This array could therefore contain up to 2TB of content. That's a lot of RAM, and probably not practical for your purposes.

So instead we can use a generator to apply a "filter" to our content. No data is actually read until we start iterating over the result.

logfile = open("hugefile.txt","r")
entry_lines = ((line,len(line)) for line in logfile if line.startswith("ENTRY"))

Not even a single line has been read from our file yet. In fact, say we want to filter our result even further:

long_entries = ((line,length) for (line,length) in entry_lines if length > 80)

Still nothing has been read, but we've specified now two generators that will act on our data as we wish.

Lets write out our filtered lines to another file:

outfile = open("filtered.txt","a")
for entry,length in long_entries:
    outfile.write(entry)

Now we read the input file. As our for loop continues to request additional lines, the long_entries generator demands lines from the entry_lines generator, returning only those whose length is greater than 80 characters. And in turn, the entry_lines generator requests lines (filtered as indicated) from the logfile iterator, which in turn reads the file.

So instead of "pushing" data to your output function in the form of a fully-populated list, you're giving the output function a way to "pull" data only when its needed. This is in our case much more efficient, but not quite as flexible. Generators are one way, one pass; the data from the log file we've read gets immediately discarded, so we can't go back to a previous line. On the other hand, we don't have to worry about keeping data around once we're done with it.

Conspectus

Answer 5

The benefit of a generator expression is that it uses less memory since it doesn't build the whole list at once. Generator expressions are best used when the list is an intermediary, such as summing the results, or creating a dict out of the results.

For example:

sum(x*2 for x in xrange(256))

dict( (k, some_func(k)) for k in some_list_of_keys )

The advantage there is that the list isn't completely generated, and thus little memory is used (and should also be faster)

You should, though, use list comprehensions when the desired final product is a list. You are not going to save any memeory using generator expressions, since you want the generated list. You also get the benefit of being able to use any of the list functions like sorted or reversed.

For example:

reversed( [x*2 for x in xrange(256)] )

Smelly

There is a hint put for you right in the language that generator expressions are meant to be used that way. Lose the brackets! sum(x*2 for x in xrange(256)) – Katonah

sorted and reversed work fine on any iterable, generator expressions included. – Excited

If you can use 2.7 and above, that dict() example would looks better as a dict comprehension (the PEP for that is older than then the generator expressions PEP, but took longer to land) – Telephone

The "should also be faster" part contradicts the answer by John Millikin though... – Absorb

Answer 6

When creating a generator from a mutable object (like a list) be aware that the generator will get evaluated on the state of the list at time of using the generator, not at time of the creation of the generator:

>>> mylist = ["a", "b", "c"]
>>> gen = (elem + "1" for elem in mylist)
>>> mylist.clear()
>>> for x in gen: print (x)
# nothing

If there is any chance of your list getting modified (or a mutable object inside that list) but you need the state at creation of the generator you need to use a list comprehension instead.

Meninges

And this should be the accepted answer. If your data are bigger than the available memory you should always use generators although looping over list in memory may be faster (but you don't have enough memory to do so). – Forego

Similarly, modifying the underlying list during the iteration over gen will cause unpredictable results, just like iterating over the list directly. – Tmesis

Answer 7

Python 3.7:

List comprehensions are faster.

Generators are more memory efficient.

As all others have said, if you're looking to scale infinite data, you'll need a generator eventually. For relatively static small and medium-sized jobs where speed is necessary, a list comprehension is best.

Gignac

It's not so simple. List comps are only faster in some cases. If you're using any and you anticipate an early False element, the generator can give a substantial improvement over a list comprehension. But if both will be exhausted, then list comps are generally faster. You really need to profile the application and see. – Debarath

Agreed if I’m potentially seeking/anticipating an early stop with the generator. Also agreed on needing a more exhaustive profiling for more complex projects. I was just providing for this simple example, appreciate the thoughts. – Gignac

Answer 8

Sometimes you can get away with the tee function from itertools, it returns multiple iterators for the same generator that can be used independently.

Ursas

Answer 9

I'm using the Hadoop Mincemeat module. I think this is a great example to take a note of:

import mincemeat

def mapfn(k,v):
    for w in v:
        yield 'sum',w
        #yield 'count',1


def reducefn(k,v): 
    r1=sum(v)
    r2=len(v)
    print r2
    m=r1/r2
    std=0
    for i in range(r2):
       std+=pow(abs(v[i]-m),2)  
    res=pow((std/r2),0.5)
    return r1,r2,res

Here the generator gets numbers out of a text file (as big as 15GB) and applies simple math on those numbers using Hadoop's map-reduce. If I had not used the yield function, but instead a list comprehension, it would have taken a much longer time calculating the sums and average (not to mention the space complexity).

Hadoop is a great example for using all the advantages of Generators.

Nauru

Answer 10

Some notes for built-in Python functions:

Use a generator expression if you need to exploit the short-circuiting behaviour of any or all. These functions are designed to stop iterating when the answer is known, but a list comprehension must evaluate every element before the function can be called.

For example, if we have

from time import sleep
def long_calculation(value):
    sleep(1) # for simulation purposes
    return value == 1

then any([long_calculation(x) for x in range(10)]) takes about ten seconds, as long_calculation will be called for every x. any(long_calculation(x) for x in range(10)) takes only about two seconds, since long_calculation will only be called with 0 and 1 inputs.

When any and all iterate over the list comprehension, they will still stop checking elements for truthiness once an answer is known (as soon as any finds a true result, or all finds a false one); however, this is usually trivial compared to the actual work done by the comprehension.

Generator expressions are of course more memory efficient, when it's possible to use them. List comprehensions will be slightly faster with the non-short-circuiting min, max and sum (timings for max shown here):

$ python -m timeit "max(_ for _ in range(1))"
500000 loops, best of 5: 476 nsec per loop
$ python -m timeit "max([_ for _ in range(1)])"
500000 loops, best of 5: 425 nsec per loop
$ python -m timeit "max(_ for _ in range(100))"
50000 loops, best of 5: 4.42 usec per loop
$ python -m timeit "max([_ for _ in range(100)])"
100000 loops, best of 5: 3.79 usec per loop
$ python -m timeit "max(_ for _ in range(10000))"
500 loops, best of 5: 468 usec per loop
$ python -m timeit "max([_ for _ in range(10000)])"
500 loops, best of 5: 442 usec per loop

Tmesis

Answer 11

List comprehensions are eager but generators are lazy.

In list comprehensions all objects are created right away, it takes longer to create and return the list. In generator expressions, object creation is delayed until request by next(). Upon next() generator object is created and returned immediately.

Iteration is faster in list comprehensions because objects are already created.

If you iterate all the elements in list comprehension and generator expression, time performance is about the same. Even though generator expression return generator object right away, it does not create all the elements. Everytime you iterate over a new element, it will create and return it.

But if you do not iterate through all the elements generator are more efficient. Let's say you need to create a list comprehensions that contains millions of items but you are using only 10 of them. You still have to create millions of items. You are just wasting time for making millions of calculations to create millions of items to use only 10. Or if you are making millions of api requests but end up using only 10 of them. Since generator expressions are lazy, it does not make all the calculations or api calls unless it is requested. In this case using generator expressions will be more efficient.

In list comprehensions entire collection is loaded to the memory. But generator expressions, once it returns a value to you upon your next() call, it is done with it and it does not need to store it in the memory any more. Only a single item is loaded to the memory. If you are iterating over a huge file in disk, if file is too big you might get memory issue. In this case using generator expression is more efficient.

Lon

Answer 12

There is something that I think most of the answers have missed. List comprehension basically creates a list and adds it to the stack. In cases where the list object is extremely large, your script process would be killed. A generator would be more preferred in this case as its values are not stored in memory but rather stored as a stateful function. Also speed of creation; list comprehension are slower than generator comprehension

In short; use list comprehension when the size of the obj is not excessively large else use generator comprehension

Suasion

Answer 13

For functional programming, we want to use as little indexing as possible. For this reason, If we want to continue using the elements after we take the first slice of elements, islice() is a better choice since the iterator state is saved.

from itertools import islice

def slice_and_continue(sequence):
    ret = []
    seq_i = iter(sequence) #create an iterator from the list

    seq_slice = islice(seq_i,3) #take first 3 elements and print
    for x in seq_slice: print(x),

    for x in seq_i: print(x**2), #square the rest of the numbers

slice_and_continue([1,2,3,4,5])

output: 1 2 3 16 25

Lamrert