Python decorator, self is mixed up [duplicate]

Question

I am new to Python decorators (wow, great feature!), and I have trouble getting the following to work because the self argument gets sort of mixed up.

#this is the decorator
class cacher(object):

    def __init__(self, f):
        self.f = f
        self.cache = {}

    def __call__(self, *args):
        fname = self.f.__name__
        if (fname not in self.cache):
            self.cache[fname] = self.f(self,*args)
        else:
            print "using cache"
        return self.cache[fname]

class Session(p.Session):

    def __init__(self, user, passw):
        self.pl = p.Session(user, passw)

    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

s = Session(u,p)
s.get_something()

When I run this, I get:

get_something called with self = <__main__.cacher object at 0x020870F0> 
Traceback:
...
AttributeError: 'cacher' object has no attribute 'pl'

for the line where I do self.cache[fname] = self.f(self,*args)

The problem - Obviously, the problem is that self is the cacher object instead of a Session instance, which indeed doesn't have a pl attribute. However I can't find how to solve this.

Solutions I've considered, but can't use - I thought of making the decorator class return a function instead of a value (like in section 2.1 of this article) so that self is evaluated in the right context, but that isn't possible since my decorator is implemented as a class and uses the build-in __call__ method. Then I thought to not use a class for my decorator, so that I don't need the __call__method, but I can't do that because I need to keep state between decorator calls (i.e. for keeping track of what is in the self.cache attribute).

Question - So, apart from using a global cache dictionary variable (which I didn't try, but assume will work), is there any other way to make this decorator work?

Edit: this SO question seems similar Decorating python class methods, how do I pass the instance to the decorator?

Donelu

You are aware that this way, all instances of Session will share the same cache? – Gothicism

yes, I've not yet completed the code but I roughly thought of given an extra session argument somewhere to keep separate caches . – Donelu

Answer 1

Use the descriptor protocol like this:

import functools

class cacher(object):

    def __init__(self, f):
        self.f = f
        self.cache = {}

    def __call__(self, *args):
        fname = self.f.__name__
        if (fname not in self.cache):
            self.cache[fname] = self.f(self,*args)
        else:
            print "using cache"
        return self.cache[fname]

    def __get__(self, instance, instancetype):
        """Implement the descriptor protocol to make decorating instance 
        method possible.

        """

        # Return a partial function with the first argument is the instance 
        #   of the class decorated.
        return functools.partial(self.__call__, instance)

Edit :

How it's work ?

Using the descriptor protocol in the decorator will allow us to access the method decorated with the correct instance as self, maybe some code can help better:

Now when we will do:

class Session(p.Session):
    ...

    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

equivalent to:

class Session(p.Session):
    ...

    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

    get_something = cacher(get_something)

So now get_something is an instance of cacher . so when we will call the method get_something it will be translated to this (because of the descriptor protocol):

session = Session()
session.get_something  
#  <==> 
session.get_something.__get__(get_something, session, <type ..>)
# N.B: get_something is an instance of cacher class.

and because :

session.get_something.__get__(get_something, session, <type ..>)
# return
get_something.__call__(session, ...) # the partial function.

so

session.get_something(*args)
# <==>
get_something.__call__(session, *args)

Hopefully this will explain how it work :)

Unpleasantness

This indeed does it, although I still have to wrap my brain around the description in the link you provided to understand what is exactly going on. I guess that self (in f(self, *args)) is resolved through the automatic __get__ call. But doesn't that provide wrong resolution for self in all other cases, e.g. self.cache[fname] and even self.f? – Donelu

@Rabarberski: i just edited my answer to include an explanation , hopefully it will be useful :) – Unpleasantness

A better way to implement __get__ is to use types.MethodType to make an actual bound (or in some cases on Py2, unbound) method object. On Py2, you'd make the __get__ body return types.MethodType(self, instance, instancetype); on Py3, you'd first test for None on instance to avoid binding (if instance is None: return self), and otherwise, you'd return types.MethodType(self, instance). Full example here. – Catechu

Answer 2

Closures are often a better way to go, since you don't have to muck about with the descriptor protocol. Saving mutable state across calls is even easier than with a class, since you just stick the mutable object in the containing scope (references to immutable objects can be handled either via the nonlocal keyword, or by stashing them in a mutable object like a single-entry list).

#this is the decorator
from functools import wraps
def cacher(f):
    # No point using a dict, since we only ever cache one value
    # If you meant to create cache entries for different arguments
    # check the memoise decorator linked in other answers
    print("cacher called")
    cache = []
    @wraps(f)
    def wrapped(*args, **kwds):
        print ("wrapped called")
        if not cache:
            print("calculating and caching result")
            cache.append(f(*args, **kwds))
        return cache[0]
    return wrapped

class C:
    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self

C().get_something()
C().get_something()

If you aren't completely familiar with the way closures work, adding more print statements (as I have above) can be illustrative. You will see that cacher is only called as the function is defined, but wrapped is called each time the method is called.

This does highlight how you need to be careful with memoisation techniques and instance methods though - if you aren't careful to account for changes in the value of self, you will end up sharing cached answers across instances, which may not be what you want.

Making

Hmm, I don't get it. The cache variable content will not be preserved between calls, no? I've looked up wraps, but I think I suffer from information overload, with the meaning and usefulness of partial() (from a previous answer) slowly trickling through. Finally, the dict is necessary because you want a different cache (key) for each function you apply cacher to, no? – Donelu

No, because cache is created when cacher is called, which only happens at function definition time. wrapped is created at the same time, and every call of wrapped will see that same original cache definition. wraps() is just a helper to copy __name__, __doc__ and other details from f to wrapped so the decorated function looks more like the original. – Making

This explanation as to how decorators work in general may also be helpful: #5482239 – Making

Answer 3

First, you explicitly pass cacher object as first argument in the following line:

self.cache[fname] = self.f(self,*args)

Python automatically adds self to the list of arguments for methods only. It converts functions (but not other callables as your cacher object!) defined in class namespace to methods. To get such behavior I see two ways:

1. Change your decorator to return function by using closures.
2. Implement descriptor protocol to pass self argument yourself as it's done in memoize decorator recipe.

Rienzi

I think solution (1) is not an option for the arguments I've given in my post. Or I am wrong about this? Solution (2) seems identical to singularity's answer. Thanks for the memoize link! – Donelu

The only argument I see is keeping state between calls. Sure, it's possible via mutable argument[s] to decorator function. – Rienzi