Behaviour of Python's "yield"

Question

I'm reading about the yield keyword in python, and trying to understand running this sample:

def countfrom(n):
    while True:
        print "before yield"
        yield n
        n += 1
        print "after yield"

for i in countfrom(10):
    print "enter for loop"
    if i <= 20:
        print i
    else:
        break

The output is:

before yield
enter for loop
10
after yield
before yield
enter for loop
11
after yield
before yield
enter for loop
12
after yield
before yield
enter for loop
13
after yield
before yield
enter for loop
14
after yield
before yield
enter for loop
15
after yield
before yield
enter for loop
16
after yield
before yield
enter for loop
17
after yield
before yield
enter for loop
18
after yield
before yield
enter for loop
19
after yield
before yield
enter for loop
20
after yield
before yield
enter for loop

It looks like the yield will return the specified value, and will continue runnning the function till the end (in a parallel thread, maybe). Is my understand correct?

If you could answer this without mentioning "generators", I would be thankful, because I'm trying to understand one at a time.

Relative

See #232267. Explaining yield without generators is pointless, because the primary (and in most cases, sole) purpose of yield is creating generators. – Elvinaelvira

-1: "If you could answer this without mentioning "generators"". Can't be done. – Egin

Answer 1

You can think of it as if the function which yields simply "pauses" when it comes across the yield. The next time you call it, it will resume after the yield keeping the state that it was in when it left.

Marvel

So the yield works like a return, and it will continue from that line, keeping the value, on next call? – Relative

No, you don't call the function another time in order to resume, you have to call the next() method on the returned generator object.. – Loft

@Loft but in my example there's no call to next(), so how is it resuming? – Relative

@Tom Brito, the next() is executed under the hood by the for loop. That is, an "iterable" object is just an object with a next() method that should return a value for each item it contains, and raise an exception when the last item is reached – Loft

@Tom: Explaining next would require use of the word you didn't want to hear, so I wrote "next call" instead. Sorry if that was more confusing than helping... – Marvel

Answer 2

No, there is only a single thread.

Each iteration of the for loop runs your countFrom function until it yields something, or returns. After the yield, the body of the for loop runs again and then, when a new iteration starts, the countFrom function picks up exactly where it left off and runs again until it yields (or returns).

This modified version of your example will helpfully make it clearer what path execution takes.

def countfrom(n):
    while n <= 12:
        print "before yield, n = ", n
        yield n
        n += 1
        print "after yield, n = ", n

for i in countfrom(10):
    print "enter for loop, i = ", i
    print i
    print "end of for loop iteration, i = ", i

Output

before yield, n =  10
enter for loop, i =  10
10
end of for loop iteration, i =  10
after yield, n =  11
before yield, n =  11
enter for loop, i =  11
11
end of for loop iteration, i =  11
after yield, n =  12
before yield, n =  12
enter for loop, i =  12
12
end of for loop iteration, i =  12
after yield, n =  13

Sporting

Great explanation. So the function pauses immediately at the yield statement and the rest of it will be going through on the next iterating. – Sheila

Answer 3

..you cannot explain the meaning of the yield statement without mentioning generators; it would be like trying to explain what a stone is without mentioning rock. That is: the yield statement is the one responsible to transform a normal function into a generator.

While you find it well documented here: http://docs.python.org/reference/simple_stmts.html#the-yield-statement

..the brief explaination of it is:

• When a function using the yield statement is called, it returns a "generator iterator", having a .next() method (the standard for iterable objects)
• Each time the .next() method of the generator is called (eg. by iterating the object with a for loop), the function is called until the first yield is encountered. Then the function execution is paused and a value is passed as return value of the .next() method.
• The next time .next() is called, the function execution is resumed until the next yield, etc. until the function returns something.

Some advantages in doing this are:

• less memory usage since memory is allocated just for the currently yielded value, not the whole list of returned values (as it would be by returning a list of values)
• "realtime" results return, as they are produced can be passed to the caller without waiting for the generation end (i used that to return output from a running process)

Loft

You beat me to the punch on the "explain one with the other" metaphor, although I was going to use spark plug and engine. This older article does a good job on outlining yield and generators too: ibm.com/developerworks/library/l-pycon/index.html – Lithopone

yeah, but diesel/steam/jet/.. engines don't have spark plugs; while yield and generators are mutually dependent :) Btw, I love IBM developerworks articles :) – Loft

Answer 4

The function countfrom is not run in a parallel thread. What happens here is that whenever the for-construct asks for the next value, the function will execute until it hits a yield statement. When the next value after that is required, the function resumes execution from where it left off.

And while you asked not to mention "generators", they are so intimately linked with yield that it doesn't really make sense to talk about the separately. What your countfrom function actually returns is a "generator object". It returns this object immediately after it is called, so the function body is not executed at all until something (e.g. a for-loop) requests values from the generator using its method .next().

Lh

Answer 5

the yield statement stores the value that you yield, until that function is called again. so if you call that function (with an iterator) it will run the function another time and give you the value. the point being that it knows where it left off last time

Ossetic

Ow, think I got it, the yield works like a return, and it will continue from the yield line on next call? – Relative

no, but it will know what the value of "n" was last time. it seems insignificant in this example because you could just pass n as a parameter, but it becomes very useful later on – Ossetic

sorry i think i misunderstood your comment. your thinking is correct (sorry) – Ossetic

looks like there is a bit of confusion about generators here (it took some time for me too to understand them completely): the state is kept not between multiple calls of the function, but between calls of the .next() method of the returned generator object. Of course, this is what is done by iterating over its return value.. – Loft

Answer 6

Python runs until it hits a yield and then stops and freezes execution. It's not continuing to run. It's hitting "after" on the next call to countfrom

It's easy to say that without making reference to generators but the fact is yield and generator are inextricably linked. To really understand it you've got to view them as the same topic.

It's easy to show yourself that what I (and others) have said is true by working with the generator from your example in a more manual way.

A function that yields instead of returning really returns a generator. You can then consume that generator by calling next. You are confused because your loop is taking care of all that in the background for you.

Here it is with the internals opened up:

def countfrom(n):
    while n <= 12:
        print "before yield, n = ", n
        yield n
        n += 1
        print "after yield, n = ", n


your_generator = countfrom(10)
next(your_generator)
print "see the after yield hasn't shown up yet, it's stopped at the first yield"
next(your_generator)
print "now it woke back up and printed the after... and continued through the loop until it got to back to yield"
next(your_generator)
print "rinse and repeate"

Applicator

Answer 7

Yield with and without for loop:

def f1():
    print('f1')
    yield 10
    print(f'f1 done')


def generator_with_for_loop():
    print(f'generator_with_for_loop')
    for f1_gen in f1():
        print(f'f1_gen={f1_gen}')


def generator_without_for_loop():
    print(f'\ngenerator_without_for_loop')
    gen = f1()
    print(f'f1_gen={gen}')
    print(gen.__next__())
    try:
        print(gen.__next__())
    except StopIteration:
        print('done')


if __name__ == '__main__':
    generator_with_for_loop()
    generator_without_for_loop()

"""
generator_with_for_loop
f1
f1_gen=10
f1 done

generator_without_for_loop
f1_gen=<generator object f1 at 0x7fd7201e54a0>
f1
10
f1 done
done
"""

Ronnaronnholm