What is generator.throw() good for?

Question

PEP 342 (Coroutines via Enhanced Generators) added a throw() method to generator objects, which allows the caller to raise an exception inside the generator (as if it was thrown by the yield expression).

I am wondering what the use cases for this feature are.

Spiro

Context: I'm currently working on a generator/coroutine implementation in PHP and I'm wondering whether or not I should include the throw() functionality. – Spiro

Do you want generators, or coroutines? While Python conflates the two, and you can build the former from the latter, they are different (as in, an entirely different league). – Glace

Among other things, this allows to implement @contextmanager decorator. – Exceptive

Answer 1

Let's say I use a generator to handle adding information to a database; I use this to store network-received information, and by using a generator I can do this efficiently whenever I actually receive data, and do other things otherwise.

So, my generator first opens a database connection, and every time you send it something, it'll add a row:

def add_to_database(connection_string):
    db = mydatabaselibrary.connect(connection_string)
    cursor = db.cursor()
    while True:
        row = yield
        cursor.execute('INSERT INTO mytable VALUES(?, ?, ?)', row)

That is all fine and well; every time I .send() my data it'll insert a row.

But what if my database is transactional? How do I signal this generator when to commit the data to the database? And when to abort the transaction? Moreover, it is holding an open connection to the database, maybe I sometimes want it to close that connection to reclaim resources.

This is where the .throw() method comes in; with .throw() I can raise exceptions in that method to signal certain circumstances:

def add_to_database(connection_string):
    db = mydatabaselibrary.connect(connection_string)
    cursor = db.cursor()
    try:
        while True:
            try:
                row = yield
                cursor.execute('INSERT INTO mytable VALUES(?, ?, ?)', row)
            except CommitException:
                cursor.execute('COMMIT')
            except AbortException:
                cursor.execute('ABORT')
    finally:
        cursor.execute('ABORT')
        db.close()

The .close() method on a generator does essentially the same thing; it uses the GeneratorExit exception combined with .throw() to close a running generator.

All this is an important underpinning of how coroutines work; coroutines are essentially generators, together with some additional syntax to make writing a coroutine easier and clearer. But under the hood they are still built on the same yielding, and sending. And when you are running multiple coroutines in parallel, you need a way to cleanly exit those coroutines if one of them has failed, just to name an example.

Mighell

Thanks for your answer. This is definitely an interesting use case. But I'm wondering whether this could be classified as exception-abuse. Commit and abort aren't exceptional conditions, but rather part of the usual behavior. So here exceptions are basically used as a means to change the control-flow. – Spiro

@Spiro Your point is valid for synchronous programming but you need to view this in the world of asynch programming. Imagine the above try block was much larger (call the code inside try, the general use case) and maybe even throw in a few more yield statements so the generator is entering and exitting during its general use case. The .throw() method allows us to "break out" to handle special exceptions. If you are familiar with interrupt handlers you can think of it like that. This way, no matter where in the use case, we can interrupt the flow to perform special (if not critical) operations – Laborious

@Spiro There is nothing wrong with using exceptions for control flow. – Highpowered

@NikiC: Python uses exceptions for control flow all the time: see the aforementioned GeneratorExit exception. While languages like C++ and Java encourage people to limit their use of exceptions to truly exceptional cases, Python does use them a lot more -- but usually across a defined interface. – Institutionalism

I assume throw() or close() happens in cursor.execute()? This will result in ValueError: Generator not running instead of Generator Exit regarldess of what error you put in. Why not just raise an error directly? – Digged

@Digged It is interesting that the example given for using throw() doesn't actually use throw(). The first line under the example I think is saying that close() uses throw(). You can't actually "raise" inside a generator though if you want to keep using the generator. Any raise will destroy it and you can no longer use it. – Dovetail

Answer 2

In my opinion the throw() method is useful for many reasons.

1. Symmetry: there is no strong reason for which an exceptional condition should be handled only in the caller and not also in the generator function. (Suppose that a generator reading values from a database returns a bad value, and suppose that only the caller knows that the value is bad. With the throw() method the caller can signal to the generator that there is an abnormal situation that has to be corrected.) If the generator can raise an exception, intercepted by the caller, the reverse should also be possible.

2. Flexibility: a generator function may have more than one yield statement, and the caller may not be aware of the internal state of the generator. By throwing exceptions it is possible to reset the generator to a known state, or to implement more sophisticated flow control which would be way more cumbersome with next(), send(), close() alone.

An example of resetting the internal state:

def gen():
    try:
        yield 10
        print("State1")
        yield 20
        print("State2")
        yield 30
        print("State3")
    
   except:
        #Reset back to State1!
        yield gen()

g = gen()
print(next(g))
print(next(g))
g = g.throw(ValueError) #state of g has been reset
print(next(g))

>>10
>>State1
>>20
>>10

Asking for use cases may be misleading: for every use case you could produce a counter example without the need for a throw() method, and the discussion would continue forever...

Henriettahenriette

can an example be given where throw is used to reset the generator to a known state? – Digged

Answer 3

One use case is to include information about the internal state of a generator in the stack trace when an exception occurs -- information that would not otherwise be visible to the caller.

For example, say we have a generator like the following where the internal state we want is the current index number of the generator:

def gen_items():
    for i, item in enumerate(["", "foo", "", "foo", "bad"]):
        if not item:
            continue
        try:
            yield item
        except Exception:
            raise Exception("error during index: %d" % i)

The following code is not sufficient to trigger the additional exception handling:

# Stack trace includes only: "ValueError: bad value"
for item in gen_items():
    if item == "bad":
        raise ValueError("bad value")

However, the following code does provide the internal state:

# Stack trace also includes: "Exception: error during index: 4"
gen = item_generator()
for item in gen:
    if item == "bad":
        gen.throw(ValueError, "bad value")

Jolson

Answer 4

This "answer" is more like a trivia.

We can (ab)use the generator's throw() to raise Exception inside a lambda, which does not otherwise support the raise statement.

foo = lambda: (_ for _ in ()).throw(Exception('foobar'))

Quoted from https://mcmap.net/q/125970/-define-a-lambda-expression-that-raises-an-exception

Rademacher

Answer 5

I'm using it to write reuseable library code that can have both synchronous and asyncio code paths. It is simplified down to something like this:

from abc import ABCMeta, abstractmethod
from typing import Generator

class Helper( metaclass = ABCMeta ):
    @abstractmethod
    def help( self, con: DatabaseConnection ) -> None:
        raise NotImplementedError
    @abstractmethod
    async def ahelp( self, con: AsyncDataConnection ) -> None:
        raise NotImplementedError

class HelperSelect( Helper ):
    ' logic here to execute a select query against the database '
    rows: list[dict[str,Any]] # help() and ahelp() write their results here

    def help( self, con: DatabaseConnection ) -> None:
        assert False, 'TODO FIXME write database execution logic here'

    async def ahelp( self, con: AsyncDataConnection ) -> None:
        assert False, 'TODO FIXME write database execution logic here'

def _application_logic() -> Generator[Helper,None,int]:
    sql = 'select * from foo'
    helper = HelperSelect( sql )
    yield helper
    # do something with helper.rows
    return 0

def do_something( con: DatabaseConnection ):
    gen = _application_logic()
    try:
        while True:
            helper = next( gen )
            try:
                helper.help( con )
            except Exception as e:
                gen.throw( e )
    except StopIteration as e:
        return e.value

async def ado_something( con: AsyncDatabaseConnection ):
    gen = _application_logic()
    try:
        while True:
            helper = next( gen )
            try:
                await helper.ahelp( con )
            except Exception as e:
                gen.throw( e )
    except StopIteration as e:
        return e.value

Without the use of gen.throw, your stack trace won't show where inside the logic the exception happened, which can be very frustrating to troubleshoot. Using gen.throw() as in the example above fixes that.

The reason for the Helper classes is because I may have a half dozen different kinds of things the logic may need to request that require asyncio besides just database queries.

I took that psuedo code and built a version that you can actually run and see the differences:

from abc import ABCMeta, abstractmethod
import logging
from typing import Any, Generator

class Helper( metaclass = ABCMeta ):
    @abstractmethod
    def help( self ) -> None:
        raise NotImplementedError

class HelperBoom( Helper ):
    def help( self ) -> None:
        assert False

def _application_logic() -> Generator[Helper,None,int]:
    helper = HelperBoom()
    yield helper
    return 0

def do_something1():
    gen = _application_logic()
    try:
        while True:
            helper = next( gen )
            helper.help()
    except StopIteration as e:
        return e.value

def do_something2():
    gen = _application_logic()
    try:
        while True:
            helper = next( gen )
            try:
                helper.help()
            except Exception as e:
                gen.throw( e )
    except StopIteration as e:
        return e.value

try:
    do_something1()
except Exception:
    logging.exception( 'do_something1 failed:' )
try:
    do_something2()
except Exception:
    logging.exception( 'do_something2 failed:' )

Here's the output, notice that do_something1's stack trace is missing the line in _application_logic, but do_something2's stack trace has that entry

ERROR:root:do_something1 failed:
Traceback (most recent call last):
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 35, in <module>
    do_something1()
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 23, in do_something1
    helper.help()
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 12, in help
    assert False
AssertionError
ERROR:root:do_something2 failed:
Traceback (most recent call last):
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 39, in <module>
    do_something2()
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 32, in do_something2
    gen.throw( e )
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 16, in _application_logic
    yield helper
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 30, in do_something2
    helper.help()
  File "C:\cvs\itas\incpy\test_helper_exceptions.py", line 12, in help
    assert False
AssertionError

Matchlock