Is there any way to override the double-underscore (magic) methods of arbitrary objects in Python?

Question

I want to write a wrapper class which takes a value and behaves just like it except for adding a 'reason' attribute. I had something like this in mind:

class ExplainedValue(object):
    def __init__(self, value, reason):
        self.value = value
        self.reason = reason

    def __getattribute__(self, name):
        print '__getattribute__ with %s called' % (name,)
        if name in ('__str__', '__repr__', 'reason', 'value'):
            return object.__getattribute__(self, name)
        value = object.__getattribute__(self, 'value')
        return object.__getattribute__(value, name)

    def __str__(self):
        return "ExplainedValue(%s, %s)" % (
            str(self.value),
            self.reason)
    __repr__ = __str__

However, the double-underscore functions don't seem to be captured with __getattribute__, for example:

>>> numbers = ExplainedValue([1, 2, 3, 4], "it worked")
>>> numbers[0]

Traceback (most recent call last):
  File "<pyshell#118>", line 1, in <module>
    numbers[0]
TypeError: 'ExplainedValue' object does not support indexing
>>> list(numbers)
__getattribute__ with __class__ called

Traceback (most recent call last):
  File "<pyshell#119>", line 1, in <module>
    list(numbers)
TypeError: 'ExplainedValue' object is not iterable

I would think the two above should end up doing this:

>>> numbers.value[0]
__getattribute__ with value called
1

>>> list(numbers.value)
__getattribute__ with value called
[1, 2, 3, 4]

Why is this not happening? How can I make it happen? (This might be a horrible idea to actually use in real code but I'm curious about the technical issue now.)

Sharilyn

I think it's an optimisation. I seem to vaguely recall from having poked at the Python source that the interpreter's C structure that backs an object (or maybe a type) has special fields for __magic__ functions. These are set when the object (or the class) is created. Whenever the interpreter internally needs to call those functions it can call it directly instead of doing a lookup through the getattr mechanism. (Obviously you should verify this by downloading the source and looking for where this structure is defined.) – Devotee

If all you want is to set an additional attribute, there's no need for a wrapper class, you can set the attribute directly on the original object: original_object.reason="because". But you probably know this already... – Recipience

Actually, straight from the horse's mouth: docs.python.org/2/reference/… - this behaviour and the rationale behind this is documented, it's not just an optimisation. (The docs for __getattribute__ also link to this.) – Devotee

@Nick I think the point of the question is why __getattribute__() doesn't get called (and log a message before the TypeError) when __getitem__() is being looked up. – Devotee

@goncalopp: you can't do that for built-ins like booleans – Sharilyn

Answer 1

As millimoose says, an implicit __foo__ call never goes through __getattribute__. The only thing you can do is actually add the appropriate functions to your wrapper class.

class Wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped

    for dunder in ('__add__', '__sub__', '__len__', ...):
        locals()[dunder] = lambda self, __f=dunder, *args, **kwargs: getattr(self.wrapped, __f)(*args, **kwargs)

obj = [1,2,3]
w = Wrapper(obj)
print len(w)

Class bodies are executed code like any other block (well, except def); you can put loops and whatever else you want inside. They're only magical in that the entire local scope is passed to type() at the end of the block to create the class.

This is, perhaps, the only case where assigning to locals() is even remotely useful.

Shoreward

For extra credit, you could write a metaclass or factory function that creates the necessary methods automatically, given the type of the object you're wrapping. – Sawyor

that might be magic too far, but note that it is a good idea for methods like __hash__, where Python does something special based on whether it exists at all. (but maybe you shouldn't be proxying __hash__ anyway.) – Shoreward

bummer! i've seen behind some of python's magic voodoo pixie dust and i am not so happy at what i've found =(. i'll live though – Sharilyn

the beauty of python's magic voodoo pixie dust is that it's all only slightly magical—except for the syntax, you could recreate all of it in python itself :) – Shoreward

Answer 2

For the sake of posterity, this is what I came up with:

class BaseExplainedValue(object):
    def __init__(self, value, reason):
        self.value = value
        self.reason = reason

    def __getattribute__(self, name):
        if name in ('value', 'reason'):
            return object.__getattribute__(self, name)
        value = object.__getattribute__(self, 'value')
        return object.__getattribute__(value, name)

    def __str__(self):
        return "<'%s' explained by '%s'>" % (
            str(self.value),
            str(self.reason))
    def __unicode__(self):
        return u"<'%s' explained by '%s'>" % (
            unicode(self.value),
            unicode(self.reason))
    def __repr__(self):
        return "ExplainedValue(%s, %s)" % (
            repr(self.value),
            repr(self.reason))

force_special_methods = set(
    "__%s__" % name for name in (
        'lt le eq ne gt ge cmp rcmp nonzero call len getitem setitem delitem iter reversed contains getslice setslice delslice' + \
        'add sub mul floordiv mod divmod pow lshift rshift and xor or div truediv' + \
        'radd rsub rmul rdiv rtruediv rfloordiv rmod rdivmod rpow rlshift rrshift rand rxor ror' + \
        'iadd isub imul idiv itruediv ifloordiv imod ipow ilshift irshift iand ixor ior' + \
        'neg pos abs invert complex int long float oct hex index coerce' + \
        'enter exit').split(),
)

def make_special_method_wrapper(method_name):
    def wrapper(self, *args, **kwargs):
        return getattr(self, method_name)(*args, **kwargs)
    wrapper.__name__ = method_name
    return wrapper

def EXP(obj, reason="no reason provided"):
    if isinstance(obj, BaseExplainedValue):
        return obj

    class ThisExplainedValue(BaseExplainedValue):
        pass
    #special-case the 'special' (underscore) methods we want
    obj_class = obj.__class__
    for method_name in dir(obj_class):
        if not (method_name.startswith("__") and method_name.endswith("__")): continue
        method = getattr(obj_class, method_name)
        if method_name in force_special_methods:
            setattr(ThisExplainedValue, method_name, make_special_method_wrapper(method_name))

    ThisExplainedValue.__name__ = "%sExplainedValue" % (obj_class.__name__,)
    return ThisExplainedValue(obj, reason)

Usage:

>>> success = EXP(True, "it went ok")
>>> if success:
        print 'we did it!'


we did it!
>>> success = EXP(False, "Server was on fire")
>>> if not success:
        print "We failed: %s" % (EXP(success).reason,)


We failed: Server was on fire

The explained values can be used interchangeably with those which they wrap:

>>> numbers = EXP([1, 2, 3, 4, 5], "method worked ok")
>>> numbers
ExplainedValue([1, 2, 3, 4, 5], 'method worked ok')
>>> numbers[3]
4
>>> del numbers[3]
>>> numbers
ExplainedValue([1, 2, 3, 5], 'method worked ok')

It even fools isinstance (explanation here):

>>> isinstance(EXP(False), bool)
True
>>> isinstance(EXP([]), list)
True

Sharilyn