Create a "with" block on several context managers? [duplicate]

Question

Suppose you have three objects you acquire via context manager, for instance A lock, a db connection and an ip socket. You can acquire them by:

with lock:
   with db_con:
       with socket:
            #do stuff

But is there a way to do it in one block? something like

with lock,db_con,socket:
   #do stuff

Furthermore, is it possible, given an array of unknown length of objects that have context managers, is it possible to somehow do:

a=[lock1, lock2, lock3, db_con1, socket, db_con2]
with a as res:
    #now all objects in array are acquired

If the answer is "no", is it because the need for such a feature implies bad design, or maybe I should suggest it in a pep? :-P

Craving

I've started a meta discussion about reversing the duplicate target for this question. – Lamination

@timgeb I've started a discussion about this duplicate closure. My apologies for not @-notifying you sooner; it slipped my mind. – Lamination

Answer 1

In Python 2.7 and 3.1 and above, you can write:

with A() as X, B() as Y, C() as Z:
    do_something()

This is normally the best method to use, but if you have an unknown-length list of context managers you'll need one of the below methods.

---

In Python 3.3, you can enter an unknown-length list of context managers by using contextlib.ExitStack:

with ExitStack() as stack:
    for mgr in ctx_managers:
        stack.enter_context(mgr)
    # ...

This allows you to create the context managers as you are adding them to the ExitStack, which prevents the possible problem with contextlib.nested (mentioned below).

contextlib2 provides a backport of ExitStack for Python 2.6 and 2.7.

---

In Python 2.6 and below, you can use contextlib.nested:

from contextlib import nested

with nested(A(), B(), C()) as (X, Y, Z):
    do_something()

is equivalent to:

m1, m2, m3 = A(), B(), C()
with m1 as X:
    with m2 as Y:
        with m3 as Z:
            do_something()

Note that this isn't exactly the same as normally using nested with, because A(), B(), and C() will all be called initially, before entering the context managers. This will not work correctly if one of these functions raises an exception.

contextlib.nested is deprecated in newer Python versions in favor of the above methods.

Lectureship

One issue: Using the simple "with open(A) as a, open(B) as b:" style syntax, line breaks make pep8 compliance as reported by standard tools seemingly impossible. I use a backslash to signify the line break, because surrounding the comma-separated expressions with parentheses results in the report of a syntax error. With the backslash, I get an "E127 continuation line over-indented" warning. I have yet to find a way to use this syntax while suppressing all warnings. – Ernestineernesto

@DarrenRinger I had this same problem. I, too, used backslashes, which I abhor, to accomplish this. Double indent all the context managers after the initial with line. Only single indent the content being wrapped. It passes flake8 for me. – Blackheart

@MartijnPieters Importantly, though, the nested call won't call the __enter__ methods of any of the context managers if B() or C() raise an exception. That could be a desirable difference. This is a good reason to keep any side effects out of the initializers and put it in __enter__, which is what you should be doing anyway. I'd argue breaking that convention is un-Pythonic. As long as the context managers follow this convention, there's no risk involved. (interjay: I'd recommend incorporating that detail into the answer, or I could do so if you don't mind.) – Waylan

@Waylan Unfortunately, this won't help with file objects, which are probably the most common use for the with statement. And the same will be true for other similar objects for which using the with statement is optional. Also, there is another problem with nested mentioned in the documentation: "if the __enter__() method of one of the inner context managers raises an exception that is caught and suppressed by the __exit__() method of one of the outer context managers, this construct will raise RuntimeError rather than skipping the body of the with statement." – Lectureship

As noted in @sage88's answer, from Python3.10 you can use parens around the sequence of context managers. – Ecotype

Answer 2

Starting in python 3.10, you'll be able to use parenthesized context managers! Thanks @iforapsy!

with (
    mock.patch('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') as a,
    mock.patch('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb') as b,
    mock.patch('cccccccccccccccccccccccccccccccccccccccccc') as c,
):
    do_something()

---

For python versions < 3.10

@interjay's Answer is correct. However, if you need to do this for long context managers, for example mock.patch context managers, then you quickly realize you want to break this across lines. Turns out you can't wrap them in parens, so you have to use backslashes. Here's what that looks like:

with mock.patch('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') as a, \
        mock.patch('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb') as b, \
        mock.patch('cccccccccccccccccccccccccccccccccccccccccc') as c:
    do_something()

Blackheart

ugly. Why not allow wrapping in parens (I got bit by this) – Fca

@JohnDeighan I agree, it is ugly. I also got bit by it, which is why I posted this. I hope this was helpful, though I agree, I wish it supported paren wrapping. – Blackheart

@JohnDeighan Why are line breaks with \ uglier than using parens? – Boren

To avoid the backslashes you can format it like this (had to link it as an image, as comments don't allow line breaks and this question is closed for additional answers). – Tea

mock.patch should evaluate to an object. can we also write a_ctx = mock.patch('a'); ...; with a_ctx as a, ...: – Surefooted

Good news, folks. Parenthesized context managers will be valid syntax starting in Python 3.10, thanks to a new parser. – Stylistic

@Stylistic fantastic, I'll look into it and update my answer accordingly. – Blackheart

I wish it would have separated the methods and target values - you could then assign all context managers to a single tuple. something like (TL;DR: this won't work): with (func(), func(), func()) as contexts: – Maze

with (*a) still doesn't work. – Beginner

As I understand, it appears to be possible (At least in 3.11) to assign the context managers to variables before the with statement, and then just go with x, y, z: - and that appears to next them. Is there something I'm missing that makes that incompatible with older versions of Python? – Revels

(In particular, I'm using this with files via x = open("filename.extension", "mode"), in case that's relevant to older versions.) – Revels

Answer 3

The first part of your question is possible in Python 3.1.

With more than one item, the context managers are processed as if multiple with statements were nested:

with A() as a, B() as b:
    suite

is equivalent to

with A() as a:
    with B() as b:
        suite

Changed in version 3.1: Support for multiple context expressions

Sazerac

thanks! but that still didn't answer my whole question: what about the 2nd case I mentioned, where the context managers are given in an array, without knowing how many mangers are there in the array. will it be possible in some python3.X to do with [cm1,cm2,cm3,cm4,cm5] as result: .... – Craving

@noam: To solve the second part of your question you could write a class to wrap a number of resources and implement __enter__ and __exit__ for that class. I'm not sure if there's a standard library class that does this already. – Sazerac

@Mark I don't think it is that easy - that's why contextlib.nested() is deprecated. If something happens between the generation of the other things and the activation of the context manager, it might happen that the cleanup doesn't happen as wanted. – Caundra

Answer 4

The second part of your question is solved with contextlib.ExitStack in Python 3.3.

Ambroid