Does Python's `all` function use short circuit evaluation?

Question

I wish to use the Python all() function to help me compute something, but this something could take substantially longer if the all() does not evaluate as soon as it hits a False. I'm thinking it probably is short-circuit evaluated, but I just wanted to make sure. Also, is there a way to tell in Python how the function gets evaluated?

---

_{Editor's note: Because any and all are functions, their arguments must be evaluated before they are called. That often creates the impression of no short-circuiting - but they do still short-circuit.}

_{To make sure that the short-circuiting can be effective, pass a generator expression, or other lazily evaluated expression, rather than a sequence. See Lazy function evaluation in any() / all() for details. Similarly, to force evaluation up-front, build a list or tuple explicitly; see How to prevent short-circuit evaluation? .}

Supersaturate

I reversed the direction of duplicate closure because this version of the question is much more accessible. The previous duplicate question, stackoverflow.com/questions/14730046, dives straight into Python test code and ended up as the basis for a bug report against Python's test suite. – Montmartre

Answer 1

Yes, it short-circuits:

>>> def test():
...     yield True
...     print('one')
...     yield False
...     print('two')
...     yield True
...     print('three')
...
>>> all(test())
one
False

From the docs:

Return True if all elements of the iterable are true (or if the iterable is empty). Equivalent to:

def all(iterable):
    for element in iterable:
        if not element:
            return False
    return True

So when it returns False, then the function immediately breaks.

Constitutionalism

What does "yield" do? – Supersaturate

@SylvesterVLowell See The Python yield keyword explained – Brayton

@SylvesterVLowell Have a look here. Simply, it's like a return statement, but you can return multiple values. But instead of returning a list, it returns a generator. – Constitutionalism

In short, yield is how generators work. If you knew that all might short circuit, I guess you knew what generators do, just not how they work. – Spectroheliograph

The part that guarantees short circuiting is here: hg.python.org/cpython/rev/124237eb5de9 – Gryphon

Answer 2

Yes, all does use short-circuit evaluation. For example:

all(1.0/x < 0.5  for x in [4, 8, 1, 0])
=> False

The above stops when x reaches 1 in the list, when the condition becomes false. If all weren't short-circuiting, we'd get a division by zero when x reached 0.

Lamdin

Answer 3

Make sure you don't do as I did initially which was to try to use short-circuiting to test for the existence of a method before calling it:

>>> class MyClass(object):
...    pass
... 
>>> a = MyClass()
>>> all([hasattr(a, 'b'), a.b()])
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
AttributeError: 'MyClass' object has no attribute 'b'

but

>>> hasattr(a, 'b') and a.b()  # doesn't evaluate a.b() as hasattr(a, 'b') is false
False

In the first code snippet, Python evaluates the list before passing it to all() so it still throws the exception. This is basically the same as using list() to force all() not to use short-circuit evaluation as in morningstar's answer

Newhouse

Python would have to have REALLY lazy evaluation to do what you initially expected. – Andy

I tried to write an answer from scratch that explains the behaviour more clearly. – Montmartre

Answer 4

In answer to your question of whether you can tell all to be either short-circuit evaluated or not, it is short-circuit by default, but if you wanted it not to be, you could do this:

result = all(list(iterable))

Though that has the possibly undesirable property that the whole list will be loaded into memory. I can't think how you would avoid that other than using a different function than all. For example

result = reduce(lambda x,y: x and y, iterable)
result = min(iterable) # surprisingly similar to all; YMMV if iterable contains non-booleans

Spectroheliograph

Don't use min() as a replacement for all() (or max() as a replacement for any()). all() will return False as soon as it encounters a falsey object; min() will always search the entire iterable to see if there's a “lesser” one. any() and max() work similarly. Also, the four of them work on any iterable—don't waste time and space making a list. – Andy

That eagerly evaluates the iterable, but the actual boolean checking logic still short-circuits. Not that it will usually matter. – Montmartre

"if you wanted it not to be" - Why would you want that? I can't think of any situation off the top of my head. – Unskillful

@wjandrea, I can't think of a specific example, but if someone wanted to call a function with desired side effects, that returned a boolean, on every item in some iterable, and also compute all() on those booleans, I could imagine someone not familiar with short-circuiting using a generator expression inside the call to all and not understanding why the function wasn't called on every item in the iterable. – Whine