Python 3: send method of generators

Question

I can't understand the send method. I understand that it is used to operate the generator. But the syntax is here: generator.send(value).

I somehow can't catch why the value should become the result of the current yield expression. I prepared an example:

def gen():
    for i in range(10):
        X = yield i
        if X == 'stop':
            break
        print("Inside the function " + str(X))

m = gen()
print("1 Outside the function " + str(next(m)) + '\n')
print("2 Outside the function " + str(next(m)) + '\n')
print("3 Outside the function " + str(next(m)) + '\n')
print("4 Outside the function " + str(next(m)) + '\n')
print('\n')
print("Outside the function " + str(m.send(None)) + '\n') # Start generator
print("Outside the function " + str(m.send(77)) + '\n')
print("Outside the function " + str(m.send(88)) + '\n')
#print("Outside the function " + str(m.send('stop')) + '\n')
print("Outside the function " + str(m.send(99)) + '\n')
print("Outside the function " + str(m.send(None)) + '\n')

The result is:

1 Outside the function 0

Inside the function None
2 Outside the function 1

Inside the function None
3 Outside the function 2

Inside the function None
4 Outside the function 3



Inside the function None
Outside the function 4

Inside the function 77
Outside the function 5

Inside the function 88
Outside the function 6

Inside the function 99
Outside the function 7

Inside the function None
Outside the function 8

Well, frankly speaking, it is astonishing me.

1. In the documentation we can read that when a yield statement is executed, the state of the generator is frozen and the value of expression_list is returned to next‘s caller. Well, it doesn't seem to have happened. Why can we execute if statement and print function inside gen().
2. How can I understand why X inside and outside the function differs? Ok. Let us assume that send(77) transmits 77 into m. Well, yield expression becomes 77. Then what is X = yield i? And how 77 inside the function converts into 5 when occurs outside?
3. Why the first result string doesn't reflect anything that is going on inside the generator?

Anyway, could you somehow comment on these send and yield statements?

Gasholder

Answer 1

When you use send and expression yield in a generator, you're treating it as a coroutine; a separate thread of execution that can run sequentially interleaved but not in parallel with its caller.

When the caller executes R = m.send(a), it puts the object a into the generator's input slot, transfers control to the generator, and waits for a response. The generator receives object a as the result of X = yield i, and runs until it hits another yield expression e.g. Y = yield j. Then it puts j into its output slot, transfers control back to the caller, and waits until it gets resumed again. The caller receives j as the result of R = m.send(a), and runs until it hits another S = m.send(b) statement, and so on.

R = next(m) is just the same as R = m.send(None); it's putting None into the generator's input slot, so if the generator checks the result of X = yield i then X will be None.

As a metaphor, consider a dumb waiter:

When the server gets an order from a customer, they put the pad in the dumb waiter, send it to the kitchen, and wait by the hatch for the dish:

R = kitchen.send("Ham omelette, side salad")

The chef (who's been waiting by the hatch) picks up the order, prepares the dish, yields it to the restaurant, and waits for the next order:

next_order = yield [HamOmelette(), SideSalad()]

The server (who's been waiting by the hatch) takes the dish to the customer and returns with another order, etc.

Because both the server and chef wait by the hatch after sending an order or yielding a dish, there's only one person doing anything at any one time i.e. the process is single threaded. Both sides can use normal control flow, as the generator machinery (the dumb waiter) takes care of interleaving execution.

Oration

it runs concurrently just not in parallel. – Megaron

@J.F.Sebastian I think "concurrently" means in parallel? – Oration

in programming "parallel" implies "concurrent" but the opposite in general is not true. Difference between concurrent programming and parallel programming – Megaron

This metaphor is awesome. Made my confusions go away. Thanks. – Irkutsk

This one made it click for me. – Registrant

but why is it necessary to call next() or send(None) to the chef for the first time? – Preston

@user2297550, see my answer for the detailed explanation. – Virgenvirgie

Answer 2

The most confusing part should be this line X = yield i, specially when you call send() on the generator. Actually the only thing you need to know is:

in the lexical level: next() is equal to send(None)

in the interpreter level: X = yield i equals to below lines(ORDER MATTERS):

yield i
# won't continue until next() or send() is called
# and this is also the entry point of next() or send()
X = the_input_of_send

and, the 2 lines of comment is the exact reason, why we need to call send(None) for the first time, because the generator will return i (yield i) before assign the value to X

Tranquilizer

Great reply ! Simple and easy explanation. You identified very well the weak point X = yield i. would upvote 10 times – Punchdrunk

"in the lexical level: next() is equal to send(None)". Made my week easier. – Ewe

this really helps! – Arquebus

i don't see why we need to call send(None) the first time. if the first time we send(7) then x would receive the value 7 so there seems to be no problem? – Preston

Is it correct that the first call to next(m) stops directly after evaluating the right hand side of yield i. I.e. it stops before returning any value to X? In that case I think I finally understand this. I think this should be the accepted answers. Everywhere else I look I always find convoluted examples that does not help with the understanding. – Vault

@Preston As we are able to assign a value to a variable inside a generator only when the generator is executed up to a yield statement, we need to call to g.send(None) or next(g) at first. In practice, sending a value to a generator except for None without executing the generator up to a yield statement, would result in a TypeError. – Sezen

@Sezen thanks for the explanation. Does it mean that the first g.send(None)` will cause the generator to execute up to but not including the yield and then I can g.send(7) to let X get the value 7? – Preston

@Preston No, the first call to g.send(None) includes an execution of the first yield statement. In other words, it is allowed to send a value to a generator only after the first yield statement is executed. I understand that sending a value g.send(v) is replacing an yield statement, for example, the rhs (yield x) of y = yield x. I think the answers to the following question would help you understand. https://mcmap.net/q/112114/-send-method-using-generator-still-trying-to-understand-the-send-method-and-quirky-behaviour/5277458 – Sezen

Answer 3

Note:
For simplicity, my answer is restricted to the case when the generator has at most 1 yield commands in every line.

TL;DR:

• The .send() method:

  • sends a value to the currently suspended yield command (waking it up), but
  • receives a value from the next, oncoming yield command.

• The recipient of a value sent by .send() method is the yield expression itself.
  It means that the expression yield 7

  • yields value 7, but
  • its own value, i.e. the value of the (yield 7), may be for example "hello"
    (parentheses are generally mandatory except the simplest cases) — if this yield 7 command was woken up by .send("hello") method.

---

The big picture:

The first send (with the None parameter) starts the generator instance, so it begins to execute its commands.

---

In detail:

Preface:

Let g = gen(), i.e. g is an instance of a generator iterator gen()
(the right-hand side of following pictures).

• The command next(g) behaves exactly as g.send(None), so you may use whichever you prefer.

• Sending a not-None value is only allowed if the instance g is suspended at the statement with the yield command:
  

  • Why? Because the .send() method may send a value only to the waiting (suspended) yield expression (see the point 4 in the “Step by step” section below).

  So before sending a not-None value, we must put the generator instance to such a suspended state by sending it the None value. It may be as simple as g.send(None):

• But just before g becomes suspended, it yields the value of the yield command. This yielded value becomes the return value of the .send() method:

  We probably want to use this received value or save it in a variable for the later use, so instead of the two previous pictures let's start our journey with this:

---

Step by step:

1. The first .send() starts the instance g. Instance g begins to execute its commands up to the first yield statement, which yields its value:

   It means, that in the variable from_iterator_1 will be the string "first_from_iterator".

    

2. Now, after yielding its first value, we have g in the suspended state

   which allows us sending to g something useful, other than None — e.g. the number 1.

    

3. So let's send the number 1 to g:

    

4. As g was suspended at the expression yield "first_from_iterator", the value of this expression (itself) will become 1.

   (Yes, the yield "first_from_iterator" is an expression, likewise a + b is.)

   Recall that at this moment the value "first_from_iterator" is long time ago already yielded.

    

5. The instance g then wakes up, and — in turn — the g.send() now waits for a returned value.

    

6. The previously suspended, now woken statement will be executed.
   (Before the suspension, it was not executed, it only yielded a value.) In our simple case (the woken statement is yield "first_from_iterator") there remains nothing to be performed, but what about

   • saving the received value (1) to a variable for the later use instead?

     received_1 = yield "first_from_iterator"      
     

   • or performing a more complicated computation with it instead?

     result = 3 * (yield "first_from_iterator") + 2           # result: 5
     

    

7. All consequent statements in g will be performed, but only up to the next statement with the yield command in it.

    

8. That next statement (with the yield command in it) yields a value

   which suspends g again, and wakes up the waiting .send() method (by providing it the awaited — yielded — return value).

    

9. It allows performing next commands after it:

    

10. Now we are in the same situation as in the point 2. — just before performing the (next) .send() method — so the story will be repeated.

    Note:
    It will be repeated with the same issue as in the last point of the “Preface” section above — we probably don't want to throw out the yielded value, so instead of the commands

    g.send(1)                          # Not very appropriate
    

    is better to use something as

    from_iterator_2 = g.send(1)        # Saving the 2nd yielded value
    

    (and similarly for the next g.send(2) command).

Virgenvirgie

Answer 4

def gen():
    i = 1
    while True:
        i += 1
        x = yield i
        print(x)

m = gen()
next(m)
next(m)
m.send(4)

result

None
4

look at more simplified codes above.
I think the thing leaded to your confusion is 'x = yield i' statment, this statment is not saying value accepted from send() method assgined to i then i assgined to x. Instead, value i is returned by yield statment to generator, x is assgined by send() method.One statement does two thing at same time.

Natoshanatron

Answer 5

Since you asked even for comments, consider the following case:

def lambda_maker():
    def generator():
        value = None
        while 1:
            value = yield value
            value= value[0][1]
    f = generator()
    next(f)  # skip the first None
    return f.send  # a handy lambda value: value[0][1]

Now the following two lines are equivalent:

a_list.sort(key=lambda a: a[0][1])
a_list.sort(key=lambda_maker())

(Incidentally, in the current (2018-05-26, day 1 post-GDPR ☺) CPython2 and CPython3 implementations, the second line runs faster than the first, but that's a detail related to frame object initialization overhead on every function call.)

What happens here? lambda_maker calls f=generator() and gets a generator; calling the initial next(f) starts running the generator and consumes the initial None value, and pauses at the yield line. Then it returns the bound method f.send to its caller. From this point on, every time this bound method is called, the generator.value local receives the argument of the bound method, recalculates value and then loops back yielding the current value of value and waits for the next .send to get another value.

The generator object stays in-memory and all it does in the loop is:

• yield the current result (initially None)
• receive another value (whatever someone used as argument to .send)
• re-calculate current result based on received value
• loop back

Irregularity

Having an explicit a_list would help to understand the example... – Codie