Why can't a module be a context manager (to a 'with' statement)?

Question

Suppose we have the following mod.py:

def __enter__():
    print("__enter__<")

def __exit__(*exc):
    print("__exit__< {0}".format(exc))

class cls:
    def __enter__(self):
        print("cls.__enter__<")

    def __exit__(self, *exc):
        print("cls.__exit__< {0}".format(exc))

and the following use of it:

import mod

with mod:
    pass

I get an error:

Traceback (most recent call last):
  File "./test.py", line 3, in <module>
    with mod:
AttributeError: __exit__

According to the documentation the documentation the with statement should execute as follows (I believe it fails at step 2 and therefore truncate the list):

1. The context expression (the expression given in the with_item) is evaluated to obtain a context manager.
2. The context manager’s __exit__() is loaded for later use.
3. The context manager’s __enter__() method is invoked.
4. etc...

As I've understood it there is no reason why __exit__ could not be found. Is there something I've missed that makes a module not able to work as a context manager?

Jenks

Why exactly do you want / need this capability? What's a use-case? – Lithesome

Answer 1

__exit__ is a special method, so Python looks it up on the type. The module type has no such method, which is why this fails.

See the Special method lookup section of the Python datamodel documentation:

For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

Note that this applies to all special methods. For example, if you added a __str__ or __repr__ function to a module it'll not be called when printing the module, either.

Python does this to make sure type objects are hashable and representable too; if Python didn't do this then trying to put a class object into a dictionary would fail when a __hash__ method was defined for that class (as that method would expect an instance to be passed in for self).

Baillieu

Answer 2

You can't do it easily for the reasons stated in @Martijn Pieters answer. However with a little extra work it is possible, because the values in sys.modules don't have to be instances of the built-in module class, they can be instances of your own custom class with the special methods a context manager requires.

Here's applying that to what you want to do. Given the following mod.py:

import sys

class MyModule(object):
    def __enter__(self):
        print("__enter__<")

    def __exit__(self, *exc):
        print("__exit__> {0}".format(exc))

# replace entry in sys.modules for this module with an instance of MyModule
_ref = sys.modules[__name__]
sys.modules[__name__] = MyModule()

And the following use of it:

import mod

with mod:
    print('running within context')

Will produces this output:

__enter__<
running within context
__exit__> (None, None, None)

See this question for information about why the _ref is needed.

Lithesome

at which point you have to ask yourself why would you need to even do this? – Baillieu

@Martijn: Replacing a module with an instance of a custom class is occasionally useful because it allows doing things that are impossible to do with a regular module object—like controlling attribute access—however I tend to agree that doing it so the module can be used as a context manager might be bit of a stretch. – Lithesome

Yes, I know there are use-cases, I'm just not sure that this is one of them. – Baillieu

@Martijn: Just answering the OP's question—while simultaneously trying to avoid passing judgement without additional information regarding their motives. – Lithesome

I am nowhere near the quality of pythonista as Martijn - but this does just seem wrong to me as well. All is does is make the import line a bit shorter, but introduces 'magic' which can easily be lost. – Dov

@Tony: The only reason the it's done in two lines the code shown is because I didn't want to hardcode "modulename" into the code (because the file it is in is not named modulename.py on my system. In actual usage it can be imported just like any other: import modulename. Since that's distracting folks, I may take it out. – Lithesome

It isn't the soft import that I have issue with - I have plenty of my own code that does that to make the code flexible, data driven etc. I have an issue with inserting a class into 'sys.modules'. All it seems to do is save you the trouble of importing the class, and make any code wishing this a bit less readable – Dov

@Tony: I actually learned about this technique in one of Alex Martelli's Python Cookbooks. He said that being able to replace a module with a class instance was an intentional feature in Python's design (although not necessarily to allow using one as a context manager). As I said in an earlier, comment, I agree that using it to do this might be questionable, and mainly just wanted to show that, in fact, one can do it—but mostly just to make more folks aware of the capability. – Lithesome

What about changing the __class__ attribute of the module for a class inheriting the Module type ? – Chian

Answer 3

A softer version than the one proposed by Martineau, a little bit less polemic :

import sys

class CustomModule(sys.modules[__name__].__class__):
  """
  Custom module
  """
  def __enter__(self):
    print('enter')

  def __exit__(self, *args, **kwargs):
    print('exit')


sys.modules[__name__].__class__ = CustomModule

Instead of replacing the module (wich may cause countless problems), just replace the class by one inheriting from the original class. This way, the original module object is kept, there is no need for another ref (preventing garbage collection), and it will work with any custom importer. Note the important fact that a module object is created and added to the sys.modules BEFORE the code of the module is executed.

Note that using this way, you can add any magic method

Chian

Clever — I like how it avoids the need for an additional reference that my answer requires. – Lithesome