How can I intercept calls to python's "magic" methods in new style classes?

Question

I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:

class ShowMeList(object):
    def __init__(self, it):
        self._data = list(it)

    def __getattr__(self, name):
        attr = object.__getattribute__(self._data, name)
        if callable(attr):
            def wrapper(*a, **kw):
                print "before the call"
                result = attr(*a, **kw)
                print "after the call"
                return result
            return wrapper
        return attr

If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.

>>> l = ShowMeList(range(8))

>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>

>>> l.append(9)
before the call
after the call

>> len(l._data)
9

If I don't inherit from object (first line class ShowMeList:) everything works as expected:

>>> l = ShowMeList(range(8))

>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]

>>> l.append(9)
before the call
after the call

>> len(l._data)
9

How do I accomplish this intercept with new style classes?

Poinsettia

What are you really trying to do by intercepting the double-underscore methods? Or is it just for curiosity? – Katharinakatharine

I always keep a list of all magic methods here: github.com/niccokunzmann/wwp/blob/… (generated. so it works for python 2 and 3) – Quadruplex

Actually it appears what you want to do is intercept calls to the magic methods of instances of a new style class -- it's still a good question IMHO, however. – Chicoine

Answer 1

For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.

class Wrapper(object):
    """Wrapper class that provides proxy access to some internal instance."""

    __wraps__  = None
    __ignore__ = "class mro new init setattr getattr getattribute"

    def __init__(self, obj):
        if self.__wraps__ is None:
            raise TypeError("base class Wrapper may not be instantiated")
        elif isinstance(obj, self.__wraps__):
            self._obj = obj
        else:
            raise ValueError("wrapped object must be of %s" % self.__wraps__)

    # provide proxy access to regular attributes of wrapped object
    def __getattr__(self, name):
        return getattr(self._obj, name)
    
    # create proxies for wrapped object's double-underscore attributes
    class __metaclass__(type):
        def __init__(cls, name, bases, dct):

            def make_proxy(name):
                def proxy(self, *args):
                    return getattr(self._obj, name)
                return proxy

            type.__init__(cls, name, bases, dct)
            if cls.__wraps__:
                ignore = set("__%s__" % n for n in cls.__ignore__.split())
                for name in dir(cls.__wraps__):
                    if name.startswith("__"):
                        if name not in ignore and name not in dct:
                            setattr(cls, name, property(make_proxy(name)))

Usage:

class DictWrapper(Wrapper):
    __wraps__ = dict

wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))

# make sure it worked....
assert "b" in wrapped_dict                        # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3)        # __eq__
assert "'a': 1" in str(wrapped_dict)              # __str__
assert wrapped_dict.__doc__.startswith("dict()")  # __doc__

Armentrout

@Armentrout I removed what looked like an orphaned line of code, but you might want to double-check that your example hasn't lost intended functionality. – Vigilant

Do you see any problem with doing ignore.update(dct) prior to looping through dir() and so condensing the last two if statements into one? Feels cleaner to me but maybe there are unintended consequences that I in my inexperience haven't anticipated. – Vigilant

On Python 3.8 I'm finding this example doesn't actually work as is. – Emilie

I've been struggling to create a proper wrapper class. This answer solved my issue entirely and with much less code than I originally thought would be necessary. – Breakage

In Python 3 you need to move the metaclass out of the class definition and pass it in to the class declaration. – Armentrout

Answer 2

Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.

Consider the following:

>>> class C:
    x = 1
    def __init__(self):
        self.y = 2
    def __getattr__(self, attr):
        print(attr)

>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z

The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).

On your example, the __repr__ and many other magic methods are already defined in the object class.

One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.

Leduc

Answer 3

As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.

Filthy

Thanks. I found this answer helpful in fleshing out your answer. – Poinsettia

Answer 4

Cut and copy from the documentation:

For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.

For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

Luxor

This answer is 100% correct while explaining nothing and being completely unhelpful. Sorry. – Martelle

An old Bell Labs joke: a pilot lost in a thick fog sees an office building, shouts out "where am I ?", gets the answer "in an airplane", and touches down in ... Allentown. How did he know ? "The answer is 100% correct ..." – Tosh