Python : Behaviour of send() in generators

Question

I was experimenting with generators in python 3 and wrote this rather contrived generator :

def send_gen():
    print("    send_gen(): will yield 1")
    x = yield 1
    print("    send_gen(): sent in '{}'".format(x))
    # yield  # causes StopIteration when left out


gen = send_gen()
print("yielded {}".format(gen.__next__()))

print("running gen.send()")
gen.send("a string")

Output:

    send_gen(): will yield 1
yielded 1
running gen.send()
    send_gen(): sent in 'a string'
Traceback (most recent call last):
  File "gen_test.py", line 12, in <module>
    gen.send("a string")
StopIteration

So gen.__next__() reaches the line x = yield 1 and yields 1. I thought x would be assigned to None, then gen.send() would look for the next yield statement because x = yield 1 is "used", then get a StopIteration.

Instead, what seems to have happened is that x gets sent "a string", which is printed, then then python attempts to look for the next yield and gets a StopIteration.

So i try this:

def send_gen():
    x = yield 1
    print("    send_gen(): sent in '{}'".format(x))


gen = send_gen()
print("yielded : {}".format(gen.send(None)))

Output :

yielded : 1

But now there's no error. send() doesn't appear to have tried to look for the next yield statement after assigning x to None.

Why is the behaviour slightly different ? Does this have to do with how I started the generators ?

Incoherent

Answer 1

The behaviour is not different; you never advanced beyond the first yield expression in the generator in the second setup. Note that StopIteration is not an error; it is normal behaviour, the expected signal to be fired whenever a generator has ended. In your second example, you just never reached the end of the generator.

Whenever a generator reaches a yield expression, execution pauses right there, the expression can't produce anything inside the generator until it is resumed. Either gen.__next__() or a gen.send() will both resume execution from that point, with the yield expression either producing the value passed in by gen.send(), or None. You could see gen.__next__() as a gen.send(None) if that helps. The one thing to realise here is that gen.send() has yield return the sent value first, and then the generator continues on to the next yield.

So, given your first example generator, this happens:

1. gen = send_gen() creates the generator object. The code is paused at the very top of the function, nothing is executed.

2. You either call gen.__next__() or gen.send(None); the generator commences and executes until the first yield expression:

   print("    send_gen(): will yield 1")
   yield 1
   

   and execution now pauses. The gen.__next__() or gen.send(None) calls now return 1, the value yielded by yield 1. Because the generator is now paused, the x = ... assignment can't yet take place! That'll only happen when the generator is resumed again.

3. You call gen.send("a string") in your first example, don't make any call in the second. So for the first example, the generator function is resumed now:

   x = <return value of the yield expression>  # 'a string' in this case
   print("    send_gen(): sent in '{}'".format(x))
   

   and now the function ends, so StopIteration is raised.

Because you never resumed the generator in your second example, the end of the generator is not reached and no StopIteration exception is raised.

Note that because a generator starts at the top of a function, there is no yield expression at that point to return whatever you sent with gen.send() so a first gen.send() value must always be None or an exception is raised. It is best to use an explicit gen.__next__() (or, rather a next(gen) function call) to 'prime' the generator so it'll be paused at the first yield expression.

Lakendra

Ahhh I see. The x = ... explanations helped a lot. Thanks ! – Incoherent

Does the execution pauses just before or after the yield assignment? I thought in x = yield 1, the yield 1 and assignment(=) were two separate steps. Please correct me if I'm wrong. – Habitable

@Habitable they are two separate steps. yield 1 pauses, until the generator is resumed, at which point the expression result is assigned to x. – Lakendra

Answer 2

The crucial difference here is that you've hit the generator in your first example twice, but you hit the generator in your second example only once.

When you define a coroutine, i.e. a generator which you intend to send arguments into, you'll have to 'prime' it beforehand by advancing to the first yield statement. Only then can you send in values. In the first example, you've done this explicitly by calling gen.__next__() before attempting to send.

In the second example, you also primed it by doing gen.send(None) (note that sending in None is actually equivalent to calling gen.__next__() or next(gen)). But then you didn't try to send in a value a second time, so there was no StopIteration in that case. The generator is just sitting there paused at the yield statement waiting for you to hit it again, and that's also why you didn't yet see the print afterwards.

Another point to note, is that if you had sent anything other than None in your second example, there would have been an error:

TypeError: can't send non-None value to a just-started generator

This is what I was talking about with 'priming' the coroutine.

Robinson

Answer 3

This is technically a co-routine and not a generator:

1. When we call send_gen(), co-routine object is created
2. When gen.___next____() is called [ We should use next(gen) ], the generator function is called which returns 1 and blocks
3. When gen.send("a string") is called, coroutine woke up and processed the input (here printed "a string")
4. Then the co-routine exited

Ries