Python multi-line with statement

Question

What is a clean way to create a multi-line with in python? I want to open up several files inside a single with, but it's far enough to the right that I want it on multiple lines. Like this:

class Dummy:
    def __enter__(self): pass
    def __exit__(self, type, value, traceback): pass

with Dummy() as a, Dummy() as b,
     Dummy() as c:
    pass

Unfortunately, that is a SyntaxError. So I tried this:

with (Dummy() as a, Dummy() as b,
      Dummy() as c):
    pass

Also a syntax error. However, this worked:

with Dummy() as a, Dummy() as b,\
     Dummy() as c:
    pass

But what if I wanted to place a comment? This does not work:

with Dummy() as a, Dummy() as b,\
     # my comment explaining why I wanted Dummy() as c\
     Dummy() as c:
    pass

Nor does any obvious variation on the placement of the \s.

Is there a clean way to create a multi-line with statement that allows comments inside it?

Acclimate

Actually, the big question is what PEP-8 says about this stuff, since PEP-8 restricts line-length to 80 chars, which is what makes doing this necessary. – Acclimate

@TigerhawkT3 I think that the 80 char limit is low too, but I see a benefit to it when I'm working on a project that requires me to have 5 files open simultaneously. It's much easier to be able to see every file. I might make an exception for this file, though. – Acclimate

PEP-8 is explicitly ok with ` \ ` line continuation for multiline with statements, since you can't use implicit continuation. That doesn't really help your situation if you want to inline comments, though. – Canterbury

Answer 1

Python 3.9+ only:

with (
    Dummy() as a,
    Dummy() as b,
    # my comment explaining why I wanted Dummy() as c
    Dummy() as c,
):
    pass

Python ≤ 3.8:

with \
    Dummy() as a, \
    Dummy() as b, \
    Dummy() as c:
    pass

Unfortunately, comments are not possible with this syntax.

Sokul

@Acclimate It works in 3.90a6, which you can install using pyenv :) – Sokul

@justin 3.9 got released – Bee

@ThomasGrainger Thanks for the ping. I tried this out and it does indeed work with Python 3.9 – Acclimate

This is not officially supported yet. It's very likely to become an official feature in 3.10, but it's currently an undocumented deviation from the official grammar. It may not work in other Python 3.9 implementations, when PyPy or some other project gets around to 3.9 support. – Jaquelinejaquelyn

@user2357112supportsMonica isn't the official grammar the one in cpython? If not, where is the official grammar? – Sokul

The official docs for the with statement disallow parentheses. It looks like when they generated the full grammar spec in the docs, they generated it from a grammar file that has the with-parentheses rule and left that rule in, though, so I guess this change is now in a semi-documented limbo. – Jaquelinejaquelyn

@user2357112supportsMonica Fair enough, this should be PR in cpython though. Guido's pretty proud of the new PEG grammar. He was quick to show me that the parenthesized with statement now works. – Sokul

@NeilG: They couldn't actually add support officially, because 3.9 officially supported being compiled with either the old LL(1) parser or the new PEG parser. 3.10 was the first release to drop LL(1) parser support, so it was the first release that could officially use syntactic features that required the PEG parser. The PEG parser is the default for 3.9, but weird/custom builds of Python 3.9 might not use it. – Ingrowth

Answer 2

As of Python 3.10, it is now possible to parenthesize the whole group of context managers, as you originally tried:

with (Dummy() as a, Dummy() as b,
      # comment about c
      Dummy() as c):
    pass

This is also technically possible in 3.9, but in a sort of semi-documented limbo.

On one hand, it's documented as new in 3.10, 3.9 wasn't supposed to introduce any features (like this one) that depend on the new parser implementation, and the 3.9 with docs forbid this form. On the other hand, the functionality ended up getting activated in the 3.9 CPython implementation, and the (mostly?) auto-generated 3.9 full grammar spec includes the parenthesized form.

---

On previous Python 3 versions, if you need to intersperse comments with your context managers, I would use a contextlib.ExitStack:

from contextlib import ExitStack

with ExitStack() as stack:
    a = stack.enter_context(Dummy()) # Relevant comment
    b = stack.enter_context(Dummy()) # Comment about b
    c = stack.enter_context(Dummy()) # Further information

This is equivalent to

with Dummy() as a, Dummy() as b, Dummy() as c:

This has the benefit that you can generate your context managers in a loop instead of needing to separately list each one. The documentation gives the example that if you want to open a bunch of files, and you have the filenames in a list, you can do

with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]

If your context managers take so much screen space that you want to put comments between them, you probably have enough to want to use some sort of loop.

---

As Mr. Deathless mentions in the comments, there's a contextlib backport on PyPI under the name contextlib2. If you're on Python 2, you can use the backport's implementation of ExitStack.

---

Incidentally, the reason you can't do something like

with (
        ThingA() as a,
        ThingB() as b):
    ...

before the new parser implementation is because a ( can also be the first token of the expression for a context manager, and CPython's old parser wouldn't be able to tell what rule it's supposed to be parsing when it sees the first (. This is one of the motivating examples for PEP 617's new PEG-based parser.

Jaquelinejaquelyn

There is a backport of contextlib improvements to Python 2 on pypi. It provides ExitStack() among other things. – C

Answer 3

This seems tidiest to me:

with open('firstfile', 'r') as (f1 # first
  ), open('secondfile', 'r') as (f2 # second
  ):
    pass

Contented

Answer 4

This isn't exactly clean, but you could do this:

with Dummy() as a, Dummy() as b, (
     #my comment
     Dummy()) as c:
    pass

There are no syntax errors, but it's not the cleanest. You could also do this:

with Dummy() as a, Dummy() as b, Dummy(
     #my comment
     ) as c:
    pass

Consider finding a way of doing this without using the comments in the middle of the with.

Acclimate

Well done Quincunx. This is fairly crazy syntax, but I think that's what OP wanted. I didn't think this was possible. – Fronton

Fun fact: he is the OP. – Contented

@Fronton This does what I want, but is there a way to do this that isn't so crazy? That's the question I want to know. Sometimes I wish python was able to say, "There's an operator at the end of this line that requires something to be after it, so I'd better check the next line for the other operand." – Acclimate

No offense intended; my amusement came from @DevShark's lack of realization. But, if you often just need to give it a few more minutes to figure something out yourself, maybe you could so before asking a question? Be more self-confident. :) – Contented

@Contented I do so before asking. I always spend at least an hour working on the problem before I ask. It's only when frustration overcomes me that I ask. And yet I somehow usually manage to figure it out after I ask. – Acclimate

@Downvoter Can you please explain what is bad about this answer so I can improve my future answers? – Acclimate

@Acclimate if you usually figure things out after finally breaking down and asking a question, might I suggest a rubber duck by your side? ;-) Either way, it's seldom a bad thing to post a question online if only to answer it yourself shortly after. It's good for posterity, and for others' benefit! – Bacardi

Answer 5

I would keep things simple and readable by adding the comment before the with statement, or on the line itself:

# my comment explaining why I wanted Dummy() as c
with Dummy() as a, Dummy() as b,\
     Dummy() as c: # or add the comment here
    pass

Apophyge

Answer 6

Like TigerhawkT3's answer, but with indenting that doesn't trigger pycodestyle's error E124:

with (
        open('firstfile', 'r')) as f1, (  # first
        open('secondfile', 'r')) as f2:  # second
    pass

IMO it's still ugly, but at least it passes the linter.

Antetype

Answer 7

To avoid too long lines, you can do the following.

Instead of:

with open(filename, 'wb') as file, request.urlopen(image.url) as response:
    pass

I, do for example:

f = lambda: open(filename, 'wb')
r = lambda: request.urlopen(image.url)
with r() as response, f() as file:
    pass

Ashil

While technically OK, most linters would raise a warning on assigning the lambda expression to a variable. See E731 do not assign a lambda expression, use a def – Wadlinger