How can I make the short-circuiting of Python's any() and all() functions effective (avoid evaluation before the function call)?

Question

Python's any and all built-in functions are supposed to short-circuit, like the logical operators or and and do.

However, suppose we have a function definition like so:

def func(s):
    print(s)
    return True

and use it to build a list of values passed to any or all:

>>> any([func('s'), func('t')])
's'
't'
True

Since the list must be constructed before any is called, the function is also evaluated ahead of time, effectively preventing the short-circuiting.

If the function calls are expensive, evaluating all the functions up front is a big loss and is a waste of this ability of any.

Knowing that any accepts any kind of iterable, how can we defer the evaluation of func, so that the short-circuiting of any prevents the call of func(t)?

Lurcher

There are all manner of iterator producing things in Python. For example map(): any(map(func, ('s', 't'))). Most people seem to have the opposite problem in Python3 — they want list outputs and python gives them generators and maps! – Darling

This has nothing to do with any. The evaluations happen while constructing the list. – Vara

Answer 1

We can use a generator expression, passing the functions and their arguments separately and evaluating only in the generator like so:

>>> any(func(arg) for arg in ('s', 't'))
's'
True

For different functions with different signatures, this could look like the following:

any(
    f(*args)
    for f, args in [(func1, ('s',)), (func2, (1, 't'))]
)

That way, any will stop iterating over the generator as soon as one function call evaluates to True, and that means that the function evaluation is fully lazy.

Another neat way to postpone the function evaluation is to use lambda expressions, like so:

>>> any(
...     f() 
...     for f in [lambda: func('s'), lambda: func('t')]
... )
's'
True

Lurcher

Answer 2

It is sad that any() and all() don't have the logical functionality and are limited by this somewhat artificial constraint. Instead, a once-thru loop construction can be handy, particularly if there are intermediate results that need to be generated and used. This is related also to a function with early returns.

def example():

    for _ in range(1):
        val = func1()
        if not val:
            break
        val1 = intermediate_func1(val)
        if not func3(val1):
            break
        val2 = intermediate_func2(val1)
        if not func4(val2):
            break
        result = the_really_expensive_function(val, val2)
        if result:
            return True, result
    return False, None

similar construct using a function and early returns.

def example():
    val = func1()
    if not val:
        return False, None
    val1 = intermediate_func1(val)
    if not func3(val1):
        return False, None
    val2 = intermediate_func2(val1)
    if not func4(val2):
        return False, None
    result = the_really_expensive_function(val, val2)
    if result:
        return True, result
    return False, None

What I wanted to use (but can't; and this is only feasible with := operator):

if all(
        val := func1(),
        func3(val1 := intermediate_func1(val)),
        func4(val2 := intermediate_func2(val1)),
        result := the_really_expensive_function(val, val2),
        ):
    return True, result
return False, None

Maybe in the future this will be feasible.

Turnspit

The first example here is missing a function definition. Could you edit to fix it? – Brook

"somewhat artificial constraint" - Eh? It's entirely a practical constraint. You can't build a list (or other collection) without knowing the contents, so the calls in OP's code have to be evaluated ahead of time. – Brook

It may be practical, but it isn't efficient. It is best not to use all() because it won't stop early, and therefore, I avoid it. I could be written to evaluate each of the all() items one at a time and not evaluate them all. It could have been written that way. – Turnspit

What you're trying to do with if all(val := func1(), func3(val1 := intermediate_func1(val)), func4(val2 := intermediate_func2(val1)), result := the_really_expensive_function(val, val2)): becomes if (val := func1()) and (func3(val1 := intermediate_func1(val))) and (func4(val2 := intermediate_func2(val1))) and (result := the_really_expensive_function(val, val2)): – Portent