How does a generator comprehension works?

Question

What does generator comprehension do? How does it work? I couldn't find a tutorial about it.

Tobe

To be clear, the language name for these is generator expressions, not generator comprehensions. – Sayles

@Sayles There is discussion on the Python-dev mailing list in July of 2018 on "Naming comprehension syntax" where there was tentative but fairly unanimous agreement to call them "generator comprehensions" for the sake of consistency. – Chinese

Answer 1

Do you understand list comprehensions? If so, a generator expression is like a list comprehension, but instead of finding all the items you're interested and packing them into list, it waits, and yields each item out of the expression, one by one.

>>> my_list = [1, 3, 5, 9, 2, 6]
>>> filtered_list = [item for item in my_list if item > 3]
>>> print(filtered_list)
[5, 9, 6]
>>> len(filtered_list)
3
>>> # compare to generator expression
... 
>>> filtered_gen = (item for item in my_list if item > 3)
>>> print(filtered_gen)  # notice it's a generator object
<generator object <genexpr> at 0x7f2ad75f89e0>
>>> len(filtered_gen) # So technically, it has no length
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'generator' has no len()
>>> # We extract each item out individually. We'll do it manually first.
... 
>>> next(filtered_gen)
5
>>> next(filtered_gen)
9
>>> next(filtered_gen)
6
>>> next(filtered_gen) # Should be all out of items and give an error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> # Yup, the generator is spent. No values for you!
... 
>>> # Let's prove it gives the same results as our list comprehension
... 
>>> filtered_gen = (item for item in my_list if item > 3)
>>> gen_to_list = list(filtered_gen)
>>> print(gen_to_list)
[5, 9, 6]
>>> filtered_list == gen_to_list
True
>>> 

Because a generator expression only has to yield one item at a time, it can lead to big savings in memory usage. Generator expressions make the most sense in scenarios where you need to take one item at a time, do a lot of calculations based on that item, and then move on to the next item. If you need more than one value, you can also use a generator expression and grab a few at a time. If you need all the values before your program proceeds, use a list comprehension instead.

Ivar

One question here. I used next(gen_name) to get the result and it worked in Python 3. Is there any specific scenario where we need to use __next__()? – Immanuel

@AnkitVashistha No, always use next(...) instead of .__next__() in Python 3. – Tonetic

@Ivar @AnkitVashistha If you need more than one value, you can also use a generator expression and grab a few at a time. Could you please give an example about this usage? Thanks. – Faxan

@KhanhLe You could do something as simple as first = next(gen) and second = next(gen) to grab yourself multiple values for use. The fantastic more-itertools package also has lots of handy tools for this, like chunked. – Recalcitrate

Answer 2

A generator comprehension is the lazy version of a list comprehension.

It is just like a list comprehension except that it returns an iterator instead of the list ie an object with a next() method that will yield the next element.

If you are not familiar with list comprehensions see here and for generators see here.

Diclinous

A generator comprehension is the lazy version of a list comprehension is probably one of the best one-liner explanations I have ever read. Thanks! – Precentor

yay! i'm glad you found it helpful! :) – Diclinous

Answer 3

List/generator comprehension is a construct which you can use to create a new list/generator from an existing one.

Let's say you want to generate the list of squares of each number from 1 to 10. You can do this in Python:

>>> [x**2 for x in range(1,11)]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

here, range(1,11) generates the list [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], but the range function is not a generator before Python 3.0, and therefore the construct I've used is a list comprehension.

If I wanted to create a generator that does the same thing, I could do it like this:

>>> (x**2 for x in xrange(1,11))
<generator object at 0x7f0a79273488>

In Python 3, however, range is a generator, so the outcome depends only on the syntax you use (square brackets or round brackets).

Welton

This is wrong. Whether the outer expression is a generator has nothing to do with whether the inner expression is. Though obviously, there's usually not much point in a generator expression taking elements from a list, you can do it. – Tbar

Can this be rewritten more clearly? I get what you are saying, but as Antimony says, it looks like you are saying something else. (and the thing it looks like you are saying is wrong) – Frameup

Answer 4

Generator comprehension is an easy way of creating generators with a certain structure. Lets say you want a generator that outputs one by one all the even numbers in your_list. If you create it by using the function style it would be like this:

def allEvens( L ):
    for number in L:
        if number % 2 is 0:
            yield number

evens = allEvens( yourList )

You could achieve the same result with this generator comprehension expression:

evens = ( number for number in your_list if number % 2 == 0 )

In both cases, when you call next(evens) you get the next even number in your_list.

Normand

Answer 5

Generator comprehension is an approach to create iterables, something like a cursor which moves on a resource. If you know mysql cursor or mongodb cursor, you may be aware of that the whole actual data never gets loaded into the memory at once, but one at a time. Your cursor moves back and forth, but there is always a one row/list element in memory.

In short, by using generators comprehension you can easily create cursors in python.

Frasier

Answer 6

Another example of Generator comprehension:

print 'Generator comprehensions'

def sq_num(n):
    for num in (x**2 for x in range(n)):    
        yield num

for x in sq_num(10):
    print x 

Berthoud

Answer 7

Generators are same as lists only, the minor difference is that in lists we get all the required numbers or items of the list at ones, but in generators the required numbers are yielded one at a time. So for getting the required items we have to use the for loop to get all the required items.

#to get all the even numbers in given range
 
def allevens(n):
    for x in range(2,n):
        if x%2==0:
            yield x

for x in allevens(10)
print(x)

#output
2
4
6
8

Trichome

Answer 8

We can understand this as a generator version of list comprehension. In the case of list comprehension, we create a using one-liners or a short code and for generator comprehensions, we do one-liner code or a small code for generators. These have the same syntax just replace the [] (square brackets) with the () curly brackets.

generator_composition_object = (num**3 for num in range(5))
print(generator_composition_object)

this will give an address of the object of the type generator. We can also use functionalities like next() in these.

Histoplasmosis