How can I decorate an instance method with a decorator class?

Question

Consider this small example:

import datetime as dt

class Timed(object):
    def __init__(self, f):
        self.func = f

    def __call__(self, *args, **kwargs):
        start = dt.datetime.now()
        ret = self.func(*args, **kwargs)
        time = dt.datetime.now() - start
        ret["time"] = time
        return ret

class Test(object):
    def __init__(self):
        super(Test, self).__init__()

    @Timed
    def decorated(self, *args, **kwargs):
        print(self)
        print(args)
        print(kwargs)
        return dict()

    def call_deco(self):
        self.decorated("Hello", world="World")

if __name__ == "__main__":
    t = Test()
    ret = t.call_deco()

which prints

Hello
()
{'world': 'World'}

Why is the self parameter (which should be the Test obj instance) not passed as first argument to the decorated function decorated?

If I do it manually, like :

def call_deco(self):
    self.decorated(self, "Hello", world="World")

it works as expected. But if I must know in advance if a function is decorated or not, it defeats the whole purpose of decorators. What is the pattern to go here, or do I misunderstood something?

Vivie

A quick google turns up this: thecodeship.com/patterns/guide-to-python-function-decorators (see the section "Decorating methods") – Cultivation

Have you read e.g. https://mcmap.net/q/206309/-decorating-class-methods-how-to-pass-the-instance-to-the-decorator/3001761, https://mcmap.net/q/354954/-python-decorator-on-instance-method/3001761 – Immaterialize

You won’t run into this kind of problem when you use a function as the decorator instead of a callable object. – Conductance

Answer 1

tl;dr

You can fix this problem by making the Timed class a descriptor and returning a partially applied function from __get__ which applies the Test object as one of the arguments, like this

class Timed(object):
    def __init__(self, f):
        self.func = f

    def __call__(self, *args, **kwargs):
        print(self)
        start = dt.datetime.now()
        ret = self.func(*args, **kwargs)
        time = dt.datetime.now() - start
        ret["time"] = time
        return ret

    def __get__(self, instance, owner):
        from functools import partial
        return partial(self.__call__, instance)

---

The actual problem

Quoting Python documentation for decorator,

The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:

def f(...):
    ...
f = staticmethod(f)

@staticmethod
def f(...):
    ...

So, when you say,

@Timed
def decorated(self, *args, **kwargs):

it is actually

decorated = Timed(decorated)

only the function object is passed to the Timed, the object to which it is actually bound is not passed on along with it. So, when you invoke it like this

ret = self.func(*args, **kwargs)

self.func will refer to the unbound function object and it is invoked with Hello as the first argument. That is why self prints as Hello.

---

How can I fix this?

Since you have no reference to the Test instance in the Timed, the only way to do this would be to convert Timed as a descriptor class. Quoting the documentation, Invoking descriptors section,

In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor.

The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.

However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead.

We can make Timed a descriptor, by simply defining a method like this

def __get__(self, instance, owner):
    ...

Here, self refers to the Timed object itself, instance refers to the actual object on which the attribute lookup is happening and owner refers to the class corresponding to the instance.

Now, when __call__ is invoked on Timed, the __get__ method will be invoked. Now, somehow, we need to pass the first argument as the instance of Test class (even before Hello). So, we create another partially applied function, whose first parameter will be the Test instance, like this

def __get__(self, instance, owner):
    from functools import partial
    return partial(self.__call__, instance)

Now, self.__call__ is a bound method (bound to Timed instance) and the second parameter to partial is the first argument to the self.__call__ call.

So, all these effectively translate like this

t.call_deco()
self.decorated("Hello", world="World")

Now self.decorated is actually Timed(decorated) (this will be referred as TimedObject from now on) object. Whenever we access it, the __get__ method defined in it will be invoked and it returns a partial function. You can confirm that like this

def call_deco(self):
    print(self.decorated)
    self.decorated("Hello", world="World")

would print

<functools.partial object at 0x7fecbc59ad60>
...

So,

self.decorated("Hello", world="World")

gets translated to

Timed.__get__(TimedObject, <Test obj>, Test.__class__)("Hello", world="World")

Since we return a partial function,

partial(TimedObject.__call__, <Test obj>)("Hello", world="World"))

which is actually

TimedObject.__call__(<Test obj>, 'Hello', world="World")

So, <Test obj> also becomes a part of *args, and when self.func is invoked, the first argument will be the <Test obj>.

Thicket

Why using a functool.partial instead of the builtin dedicated types.MethodType ? – Southerland

Thank you for this comprehensive answer - I'm going to have to read it a few times to grok it, but I'm confident that I will! When I first tried applying this, I left off the (object) in the definition of my decorator - so, I just had class Timed: - and I still got the OP's original error. Could you point me at the right concept to read up on to understand why? – Vincentia

@Vincentia - Were you in Python2.7? In 2.7 you want to make sure every class inherits from object; this inheritance is default in Python3. I don't know specifically why this manifested in a bug, but that's the likely culprit - type(YourClass) should be type, not classobj. – Anissaanita

Can you please explain why def check_authorization(f): def wrapper(*args): print(args[0].url) return f(*args) return wrapper class Client(object): def __init__(self, url): self.url = url @check_authorization def get(self): print('get') works here? wouldn't it be also unbound? – Zavras

Some great docs on the relationship between self, decorators, descriptors, functions, and methods can be found here (if you are just interested in the section on functions and methods, check here). It definitely helped by understanding. – Knucklehead

I have been struggling to get this to work when you have a decorator that takes in parameters. Do you have any thoughts on how to make this work with, for example @Timed(interval=5)? – Downy

is there a way to add arguments to Timed object so we can do something like @Timed(some_init_arg)? – Laminated

I made a Notebook that shows how one could use a Decorator with custom arguments colab.research.google.com/drive/… It does work but I am not sure if this is the best approach for solving the issue. – Duncan

I just made a copy of your drive, hope it would work colab.research.google.com/drive/… @Duncan – Gabardine

Answer 2

You first have to understand how function become methods and how self is "automagically" injected.

Once you know that, the "problem" is obvious: you are decorating the decorated function with a Timed instance - IOW, Test.decorated is a Timed instance, not a function instance - and your Timed class does not mimick the function type's implementation of the descriptor protocol. What you want looks like this:

import types

class Timed(object):
    def __init__(self, f):
        self.func = f

    def __call__(self, *args, **kwargs):
        start = dt.datetime.now()
        ret = self.func(*args, **kwargs)
        time = dt.datetime.now() - start
        ret["time"] = time
        return ret

   def __get__(self, instance, cls):           
       return types.MethodType(self, instance)

Southerland

Thanks, this helped. One note though, this doesn't work in Python 3, since unbound methods no longer exist (MethodType only takes two arguments, and the second must not be None). For Python 3 the alternative would be: return types.MethodType(self, instance) if instance else self. – Bicameral

@Dário That should probably be if instance is not None else. Don't want to trigger the else just because instance happens to be falsy. – Eaten

Indeed @DominickPastore. – Bicameral

Answer 3

I combined some answers and comments, especially from @PythonF, who had a google collab link that was pretty helpful in seeing how different methods actually work. My aim is not to be the best answer out there, because others know far better, but despite all of the other great answers, no one actually answered the question with code that is complete and reuseable, so here's some minimal code with test cases.

This can accept arguments and passes the instance in properly:

    class Decorator:
        def __init__(self, func = None, start_text = "Start", stop_text = "Stop"):
            self.func = func
            self.instance = None
            self.start_text = start_text
            self.stop_text = stop_text
    
        def __call__(self, func):
            if self.func is None:
                self.func = func
            def call(*args, **kwargs):
                if self.instance is None and len(args) > 0:
                    self.instance = args[0]
                # do stuff before
                print(f"--- {self.start_text} ---")
                wrapped_method = self.func(self.instance, *args[1:], **kwargs)
                # do stuff afterwards
                print(f"--- {self.stop_text} ---")
                return wrapped_method
            return call
    

    class HelloWorld:
        def __init__(self):
            self.test = "test"
    
        @Decorator(start_text="Starting...", stop_text="Done")
        def print(self, name):
            print(name)
            print(self.test)
            return 42


    hello_world = HelloWorld()
    hello_world.print("Max Musterman")

Factotum