__getattr__ on a module

Question

How can implement the equivalent of a __getattr__ on a class, on a module?

Example

When calling a function that does not exist in a module's statically defined attributes, I wish to create an instance of a class in that module, and invoke the method on it with the same name as failed in the attribute lookup on the module.

class A(object):
    def salutation(self, accusative):
        print "hello", accusative

# note this function is intentionally on the module, and not the class above
def __getattr__(mod, name):
    return getattr(A(), name)

if __name__ == "__main__":
    # i hope here to have my __getattr__ function above invoked, since
    # salutation does not exist in the current namespace
    salutation("world")

Which gives:

matt@stanley:~/Desktop$ python getattrmod.py 
Traceback (most recent call last):
  File "getattrmod.py", line 9, in <module>
    salutation("world")
NameError: name 'salutation' is not defined

Entopic

You aren't in your case even making an attribute access, so you are asking for two different things at once. So the major question is which one you want. Do you want salutation to exist in the global or local namespace (which is what the code above is trying to do) or do you want dynamic lookup of names when you make a dot access on a module? It's two different things. – Sofiasofie

Interesting question, how did you come up with this? – Signora

One of Python's mantras is "explicit is better than implicit". I think S.Lott's solution is the best under that reasoning. – Bandicoot

possible duplicate of Autoload in Python – Militia

may be used to deprecate module level globals stackoverflow.com/questions/922550/… – Picro

__getattr__ on modules is supported from Python 3.7 – Olivette

This works in py37 getattr(your_module,'YourClass') and getattr(your_module,'your_function') – Eburnation

Answer 1

A while ago, Guido declared that all special method lookups on new-style classes bypass __getattr__ and __getattribute__. Dunder methods had previously worked on modules - you could, for example, use a module as a context manager simply by defining __enter__ and __exit__, before those tricks broke.

Recently some historical features have made a comeback, the module __getattr__ among them, and so the existing hack (a module replacing itself with a class in sys.modules at import time) should be no longer necessary.

In Python 3.7+, you just use the one obvious way. To customize attribute access on a module, define a __getattr__ function at the module level which should accept one argument (name of attribute), and return the computed value or raise an AttributeError:

# my_module.py

def __getattr__(name: str) -> Any:
    ...

This will also allow hooks into "from" imports, i.e. you can return dynamically generated objects for statements such as from my_module import whatever.

On a related note, along with the module getattr you may also define a __dir__ function at module level to respond to dir(my_module). See PEP 562 for details.

Methodius

If I dynamically create a module via m = ModuleType("mod") and set m.__getattr__ = lambda attr: return "foo"; however, when I run from mod import foo, I get TypeError: 'module' object is not iterable. – Fort

@weberc2: Make that m.__getattr__ = lambda attr: "foo", plus you need to define an entry for the module with sys.modules['mod'] = m. Afterward, there's no error with from mod import foo. – Daedal

wim: You can also get dynamically computed values—like having a module-level property—allowing one to write my_module.whatever to invoke it (after an import my_module). – Daedal

Regarding Cython support for PEP 562: defining a module function __getattr__ and using it to access globals(), as described in the first example of PEP 562, appears to work as expected. Declaring public or read-only variables is not yet possible in a Cythonic way from what I understand, see: github.com/cython/cython/issues/656 and github.com/cython/cython/issues/3959, and also: cython.readthedocs.io/en/latest/src/changes.html (Section 0.29.22, Subsection "Other changes"). – Asteriated

Answer 2

There are two basic problems you are running into here:

1. __xxx__ methods are only looked up on the class
2. TypeError: can't set attributes of built-in/extension type 'module'

(1) means any solution would have to also keep track of which module was being examined, otherwise every module would then have the instance-substitution behavior; and (2) means that (1) isn't even possible... at least not directly.

Fortunately, sys.modules is not picky about what goes there so a wrapper will work, but only for module access (i.e. import somemodule; somemodule.salutation('world'); for same-module access you pretty much have to yank the methods from the substitution class and add them to globals() eiher with a custom method on the class (I like using .export()) or with a generic function (such as those already listed as answers). One thing to keep in mind: if the wrapper is creating a new instance each time, and the globals solution is not, you end up with subtly different behavior. Oh, and you don't get to use both at the same time -- it's one or the other.

---

Update

From Guido van Rossum:

There is actually a hack that is occasionally used and recommended: a module can define a class with the desired functionality, and then at the end, replace itself in sys.modules with an instance of that class (or with the class, if you insist, but that's generally less useful). E.g.:

# module foo.py

import sys

class Foo:
    def funct1(self, <args>): <code>
    def funct2(self, <args>): <code>

sys.modules[__name__] = Foo()

This works because the import machinery is actively enabling this hack, and as its final step pulls the actual module out of sys.modules, after loading it. (This is no accident. The hack was proposed long ago and we decided we liked enough to support it in the import machinery.)

So the established way to accomplish what you want is to create a single class in your module, and as the last act of the module replace sys.modules[__name__] with an instance of your class -- and now you can play with __getattr__/__setattr__/__getattribute__ as needed.

---

Note 1: If you use this functionality then anything else in the module, such as globals, other functions, etc., will be lost when the sys.modules assignment is made -- so make sure everything needed is inside the replacement class.

Note 2: To support from module import * you must have __all__ defined in the class; for example:

class Foo:
    def funct1(self, <args>): <code>
    def funct2(self, <args>): <code>
    __all__ = list(set(vars().keys()) - {'__module__', '__qualname__'})

Depending on your Python version, there may be other names to omit from __all__. The set() can be omitted if Python 2 compatibility is not needed.

Gilroy

This works because the import machinery is actively enabling this hack, and as its final step pulls the actual module out of sys.modules, after loading it Is it mentioned somewhere in the docs? – Militia

Now I feel more at ease using this hack, considering it "semi-sanctioned" :) – Milkman

This is doing screwy things, like making import sys give None for sys. I'm guessing this hack isn't sanctioned in Python 2. – Twiddle

@asmeurer: This hack was created in Python 2. You must be doing something wrong. ;) – Gilroy

@asmeurer: To understand the reason for that (and a solution) see the question Why is the value of __name__ changing after assignment to sys.modules[__name__]?. – Daedal

@martineau: Interesting! Now we know why reassigning sys.modules[__name__] should be the last thing one does! ;) – Gilroy

Is this still the recommended way many years later? Namely is it still the right hack in Python3.5? – Leucomaine

@qarma: I seem to recall some enhancements being talked about that would allow python modules to more directly participate in the class inheritance model, but even so this method still works and is supported. – Gilroy

As written here, it does not work properly, except for the most trivial toy examples which don't need to use any global/closure/outer names. Otherwise, you'll need to defeat the GC, for example using the reference count hack as shown in mouad's answer here. – Methodius

@wim: If one is using this advanced functionality, one should be able to put all needed globals/closures/outer names inside the class that will be replacing the module. – Gilroy

Q: It works for import foo. But is there a way to get it working also for from foo import *? – Eph

@Friedrich: Yes. Check Note 2 in my updated answer. – Gilroy

@Ethan Furman: Thanks. I'm deeply impressed by this hack. For python3, it results in TypeError: 'dict_keys' object does not support indexing. To get it working for python 2 and 3, I use __all__ = list(set(vars().keys()) - {'__qualname__'}). – Eph

@Friedrich: Thanks -- I had tested with Python 2. – Gilroy

@EthanFurman: On Python 3.4+, I think it is better to also set the __spec__ attribute of the class (e.g., add the line __spec__ = __spec__ to the class definition). – Siphonophore

Yo all! 🧨At least concerning modules, not whatever the OP wanted at the class level, this is OUTDATED🧨 See PEP-562 and Wim's answer. – Dariadarian

Answer 3

This is a hack, but you can wrap the module with a class:

class Wrapper(object):
  def __init__(self, wrapped):
    self.wrapped = wrapped
  def __getattr__(self, name):
    # Perform custom logic here
    try:
      return getattr(self.wrapped, name)
    except AttributeError:
      return 'default' # Some sensible default

sys.modules[__name__] = Wrapper(sys.modules[__name__])

Wolff

That may work but it's probably not a solution to the real problem of the author. – Tetrasyllable

"May work" and "probably not" isn't very helpful. It's a hack/trick, but it works, and solves the problem posed by the question. – Viper

While this will work in other modules that import your module and access nonexistent attributes on it, it won't work for the actual code example here. Accessing globals() does not go through sys.modules. – Danieldaniela

Unfortunately this doesn't work for the current module, or likely for stuff accessed after an import *. – Entopic

Answer 4

We don't usually do it that way.

What we do is this.

class A(object):
....

# The implicit global instance
a= A()

def salutation( *arg, **kw ):
    a.salutation( *arg, **kw )

Why? So that the implicit global instance is visible.

For examples, look at the random module, which creates an implicit global instance to slightly simplify the use cases where you want a "simple" random number generator.

Phillis

If you're really ambitious, you could create the class, and iterate through all its methods and create a module-level function for each method. – Mercerize

@Paul Fisher: Per the problem, the class already exists. Exposing all methods of the class might not be a good idea. Usually these exposed methods are "convenience" methods. Not all are appropriate for the implicit global instance. – Phillis

Answer 5

Similar to what @Håvard S proposed, in a case where I needed to implement some magic on a module (like __getattr__), I would define a new class that inherits from types.ModuleType and put that in sys.modules (probably replacing the module where my custom ModuleType was defined).

See the main __init__.py file of Werkzeug for a fairly robust implementation of this.

Intimacy

Answer 6

This is hackish, but...

# Python 2.7
import types


class A(object):
    def salutation(self, accusative):
        print("hello", accusative)
    def farewell(self, greeting, accusative):
         print(greeting, accusative)


def AddGlobalAttribute(classname, methodname):
    print("Adding " + classname + "." + methodname + "()")
    def genericFunction(*args):
        return globals()[classname]().__getattribute__(methodname)(*args)
    globals()[methodname] = genericFunction


# set up the global namespace
x = 0   # X and Y are here to add them implicitly to globals, so
y = 0   # globals does not change as we iterate over it.


toAdd = []


def isCallableMethod(classname, methodname):
    someclass = globals()[classname]()
    something = someclass.__getattribute__(methodname)
    return callable(something)


for x in globals():
    print("Looking at", x)
    if isinstance(globals()[x], (types.ClassType, type)):
        print("Found Class:", x)
        for y in dir(globals()[x]):
            if y.find("__") == -1: # hack to ignore default methods
                if isCallableMethod(x,y):
                    if y not in globals(): # don't override existing global names
                        toAdd.append((x,y))
    # Returns:
    # ('Looking at', 'A')
    # ('Found Class:', 'A')
    # ('Looking at', 'toAdd')
    # ('Looking at', '__builtins__')
    # ('Looking at', 'AddGlobalAttribute')
    # ('Looking at', 'register')
    # ('Looking at', '__package__')
    # ('Looking at', 'salutation')
    # ('Looking at', 'farewell')
    # ('Looking at', 'types')
    # ('Looking at', 'x')
    # ('Looking at', 'y')
    # ('Looking at', '__name__')
    # ('Looking at', 'isCallableMethod')
    # ('Looking at', '__doc__')
    # ('Looking at', 'codecs')



for x in toAdd:
    AddGlobalAttribute(*x)


if __name__ == "__main__":
    salutation("world")
    farewell("goodbye", "world")


# Returns:
# hello world
# goodbye world

This works by iterating over the all the objects in the global namespace. If the item is a class, it iterates over the class attributes. If the attribute is callable it adds it to the global namespace as a function.

It ignore all attributes which contain "__".

I wouldn't use this in production code, but it should get you started.

Combustion

I prefer Håvard S's answer to mine, as it appears much cleaner, but this directly answers the question as asked. – Combustion

This is a lot closer to what I eventually went with. It's a little messy, but works with globals() correctly within the same module. – Entopic

Seems to me like this answer isn't quite what was asked for which was "When calling a function that does not exist in a module's statically defined attributes" because it's doing it's work unconditionally and adding every possible class method. That could be fixed by using to a module wrapper that only does the AddGlobalAttribute() when there's a module level AttributeError -- kind of the reverse of @Håvard S's logic. If I have a chance I'll test this out and add my own hybrid answer even though the OP has accepted this answer already. – Daedal

Update to my previous comment. I now understand that it's very hard (impssoble?) to intercept NameError exceptions for the global (module) namespace -- which explains why this answer adds callables for every possiblity it finds to the current global namespace to cover every possible case ahead of time. – Daedal

Answer 7

Here's my own humble contribution -- a slight embellishment of @Håvard S's highly rated answer, but a bit more explicit (so it might be acceptable to @S.Lott, even though probably not good enough for the OP):

import sys

class A(object):
    def salutation(self, accusative):
        print "hello", accusative

class Wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped

    def __getattr__(self, name):
        try:
            return getattr(self.wrapped, name)
        except AttributeError:
            return getattr(A(), name)

_globals = sys.modules[__name__] = Wrapper(sys.modules[__name__])

if __name__ == "__main__":
    _globals.salutation("world")

Daedal

Answer 8

For a module, see PEP-562, present since 3.7:

---

(Note: after writing this up, I realized Wim's answer says just this, but of course I somehow missed it paging through all the now-obsolete answers :-(

And this is despite knowing that this feature existed from listening to the fine Python Bytes podcast.

We really, really, need a deprecation flag on answers.

Still, I hope someone will find actual working code useful.

---

I couldn't understand the PEP, so used some tests to get it working:

module1.py

from datetime import date, datetime

_OBSOLETES = {
    "oldanswer" : 24
}

datenow = date.today()

def __getattr__(name):
    if name == "timenow":
        return datetime.now()
    if (v:=_OBSOLETES.get(name)):
        print(f"😭 variable '{name}' is deprecated")
        return v

    raise AttributeError(name)

main.py

"""test getattr module"""
import module1


print(f"{module1.datenow=}")
print(f"{module1.timenow=}")

try:
    print(f"{module1.theanswer=}")
except (AttributeError,) as e: 
    print(f"❌{e=}")

print(f"{module1.oldanswer=}")

print(f"{getattr(module1,'theanswer',42)=}")

output:

module1.datenow=datetime.date(2023, 12, 11)
module1.timenow=datetime.datetime(2023, 12, 11, 22, 32, 22, 546330)
❌e=AttributeError('theanswer')
😭 variable 'oldanswer' is deprecated
module1.oldanswer=24
getattr(module1,'theanswer',42)=42

Dariadarian

Answer 9

Create your module file that has your classes. Import the module. Run getattr on the module you just imported. You can do a dynamic import using __import__ and pull the module from sys.modules.

Here's your module some_module.py:

class Foo(object):
    pass

class Bar(object):
    pass

And in another module:

import some_module

Foo = getattr(some_module, 'Foo')

Doing this dynamically:

import sys

__import__('some_module')
mod = sys.modules['some_module']
Foo = getattr(mod, 'Foo')

Ethban

You're answering a different question here. – Danieldaniela