Short-circuiting list comprehensions [duplicate]

Question

On several occasions I've wanted python syntax for short-circuiting list comprehensions or generator expressions.

Here is a simple list comprehension, and the equivalent for loop in python:

my_list = [1, 2, 3, 'potato', 4, 5]
[x for x in my_list if x != 'potato']

result = []
for element in my_list:
    if element != 'potato':
        result.append(element)

There isn't support in the language for a comprehension which short-circuits. Proposed syntax, and equivalent for loop in python:

[x for x in my_list while x != 'potato']
# --> [1, 2, 3]

result = []
for element in my_list:
    if element != 'potato':
        result.append(element)
    else:
        break

It should work with arbitrary iterables, including infinite sequences, and be extendible to generator expression syntax. I am aware of list(itertools.takewhile(lambda x: x != 'potato'), my_list) as an option, but:

• it's not particularly pythonic - not as readable as a while comprehension
• it probably can't be as efficient or fast as a CPython comprehension
• it requires an additional step to transform the output, whereas that can be put into a comprehension directly, e.g. [x.lower() for x in mylist]
• even the original author doesn't seem to like it much.

My question is, was there any theoretical wrinkle about why it's not a good idea to extend the grammar to this use case, or is it just not possible because python dev think it would be rarely useful? It seems like a simple addition to the language, and a useful feature, but I'm probably overlooking some hidden subtleties or complications.

Related: this and this

Janka

result = []; any(x=='potato' or result.append(x) for x in my_list). Hmm...no I think this is worse than takewhile :) – Medallist

I think you are more likely to get a good answer in the python-dev mailing list. – Waldon

@gnibbler: you just implemented filterfalse, OP wants to stop all appending after first 'potato' is found. Try: found = []; result = []; any(x=='potato' and not found.append(x) or result.append(x) if not found else None for x in my_list) – Cata

Another syntax suggestion: [x if x != 'potato' else break for x in my_list] – Medallist

@gnibbler's and my initial code golf comments are non-short-circuiting, though. – Cata

@PaulMcGuire, any shortcircuits it – Medallist

@gnibbler - Doh! Thank you. (either too much or not enough coffee today) – Cata

list(iter(iter(my_list).next, "potato")) sounds funny when you say it out loud – Medallist

@gnibbler the discussion in the mailing list contains some tricks you'll probably like. – Waldon

Answer 1

Turns out, as @Paul McGuire noted, that it had been proposed in PEP 3142 and got rejected by Guido:

I didn't know there was a PEP for that. I hereby reject it. No point wasting more time on it.

He doesn't give explanations, though. In the mailing list, some of the points against it are:

• "[comprehension are] a carefully designed 1 to 1 transformation between multiple nested statements and a single expression. But this proposal ignores and breaks that. (here)." That is, the while keyword does not correspond to a while in the explicit loop - it is only a break there.
• "It makes Python harder to learn because it adds one more thing to learn." (cited here)
• It adds another difference between generator expressions and list-comprehension. Seems like adding this syntax to list comprehension too is absolutely out of the question.

I think one basic difference from the usual list comprehension is that while is inherently imperative, not declarative. It depends and dictates an order of execution, which is not guaranteed by the language (AFAIK). I guess this is the reason it is not included in Haskell's comprehensions, from which Python stole the idea.

Of course, generator expressions do have direction, but their elements may be precomputed - again, AFAIK. The PEP mentioned did propose it only for generator expressions - which makes some sense.

Of course, Python is an imperative language anyway, but it will raise problems.

What about choosing out of a non-ordered collection?

[x for x in {1,2,3} while x!=2]

You don't have it in simple for loops too, but that's something you can't enforce by the language. takewhile answers this question, but it is an arbitrary answer.

---

One last point, note that for consistency you will want support for dropwhile. something like

[x for x in my_list from x != 'potato']

Which is even less related to the equivalent for construct, and this time it is not possibly short circuit if my_list is just an iterable.

Waldon

You can just as easily write the for loop version as for x in my_set:... This doesn't make much sense to do anything useful either. – Medallist

I think the absence of the dropwhile option without introducing a new keyword (like "after" in [x for x in my_list after x=='potato']) has been one of the hurdles to adoption, but I really like the idea - unfortunately, it has already been submitted as PEP 3142 (python.org/dev/peps/pep-3142) and rejected. – Cata

@gnibbler I referred to it explicitly, although not in a clear way: it is an arbitrary answer, which is better be avoided by the language. In a for loop there's simply no real way to restrict it. – Waldon

and @PaulMcGuire, what's wrong with from? – Waldon

@Waldon - would you need to say [x for x in my_list from x == 'potato']? To my ears, this would include the 'potato' entry, not just everything after 'potato'. If you really meant "!= 'potato'", then this is even more confusing to me, since entries before 'potato' are also not equal to 'potato'. Why was the PEP rejected? Check archived discussion on Python dev list (mail.python.org/pipermail/python-dev). – Cata

So it was proposed years ago, interesting! Guido must have been having a bad day, his response was a little brash. I like your dropwhile syntax, that's a very elegant use of an existing keyword. – Janka

@Janka but one that has absolutely no relation with its current meaning. - simply overloading it. (I am not saying I don't like the idea, personally). – Waldon

Or somethng like [x for x in iterable pass cond(x)] – Janka

@Janka even worse. I wouldn't understand what you mean. – Waldon

I believe it only bothers you because lambda x : x*x is such an ugly and long way to say (x => x*x). – Waldon

To be honest I have never used dropwhile, but I've needed takewhile a few times .. – Janka