Get defining class of unbound method object in Python 3
Asked Answered
P

5

58

Say I want to make a decorator for methods defined in a class. I want that decorator, when invoked, to be able to set an attribute on the class defining the method (in order to register it in a list of methods that serve a particular purpose).

In Python 2, the im_class method accomplishes this nicely:

def decorator(method):
  cls = method.im_class
  cls.foo = 'bar'
  return method

However, in Python 3, no such attribute (or a replacement for it) seems to exist. I suppose the idea was that you could call type(method.__self__) to get the class, but this does not work for unbound methods, since __self__ == None in that case.

NOTE: This question is actually a bit irrelevant for my case, since I've chosen instead to set an attribute on the method itself and then have the instance scan through all of its methods looking for that attribute at the appropriate time. I am also (currently) using Python 2.6. However, I am curious if there is any replacement for the version 2 functionality, and if not, what the rationale was for removing it completely.

EDIT: I just found this question. This makes it seem like the best solution is just to avoid it like I have. I'm still wondering why it was removed though.

Personally answered 28/8, 2010 at 3:12 Comment(7)
The removal of unbound methods is documented here: docs.python.org/py3k/whatsnew/…Bosson
Guido van Rossum's rationale for removing unbound methods can be found here: mail.python.org/pipermail/python-dev/2005-January/050625.html, and the blog mentioned in that post is here: artima.com/weblogs/viewpost.jsp?thread=86641Edrisedrock
Thanks to all of you. That was exactly what I was looking for.Personally
The solution described in the question you linked to (use a decorator to just tag the functions, and then use a class decorator to modify the tagged functions after the fact) has some nice advantages. It's explicit, it doesn't rely on anything tricky or not-well-known, it's guaranteed to work in any version of Python (well, you need decorators… but even without them, spam = deco(spam) works), it's flexible to a wide range of similar but not identical problems, …Gecko
Your example is incorrect. When you will use such decorator on a function in class it fails. Because function becomes a bound/unbound method on getting attribute.Innerve
@PavelPatrin You're right, looks like it only works as decorator(Foo.bar), not as @decorator. I was probably doing something slightly different, but it's been six years and it never worked where I wanted it anyway, so who knows.Personally
@NedDeily: slight correction: unbound method objects were removed from Python 3, but unbound methods still exist, e.g. str.splitPsychognosis
S
47

The point you appear to be missing is, in Python 3 the "unbound method" type has entirely disappeared -- a method, until and unless it's bound, is just a function, without the weird "type-checking" unbound methods used to perform. This makes the language simpler!

To wit...:

>>> class X:
...   def Y(self): pass
... 
>>> type(X.Y)
<class 'function'>

and voila -- one less subtle concept and distinction to worry about. Such simplifications are the core advantage of Python 3 wrt Python 2, which (over the years) had been accumulating so many subtleties that it was in danger (if features kept being added to it) of really losing its status as a simple language. With Python 3, simplicity is back!-)

Strained answered 28/8, 2010 at 3:20 Comment(6)
That makes sense. I didn't even realize there was extra functionality for methods like the first-argument type checking--I just thought they would be normal functions with a few extra attributes. With that, I can completely understand the removal.Personally
@Tim, yes, the check that the first argument was of an instance of the .im_class (including subclasses thereof of course) is why im_class was kept in the first place (and the same layer of indirectness wrapped around functions to make unbound methods, as needs to be wrapped anyway to make bound ones). The underlying function always was (and still is for bound methods) the .im_func, BTW, never the method object itself.Strained
"With Python 3, simplicity is back!" sounds like it would make a good bumper sticker (or laptop sticker).Cidevant
Would anyone mind explaining how this answers tim's question? Thanks!Maxilliped
@GershomMaes, if a question was, "how do I get the defining class of a unicorn", are you really unable to see how "unicorns don't exist" answers it? The question being "how do I get the defining class of an unbound method in Python 3", clearly "unbound methods in Python 3 do not exist" answers it in exactly the same way: obviously, you can't "get the defining class" (or any other characteristic) of something that just does not exist!-)Strained
Ahh I see, I didn't realize the question was particularly asking about unbound methods! ThanksMaxilliped
M
125

I thought it would be worthwhile writing something that does it best at guessing the defining class. For completeness' sake this answer also addresses bound methods.

At worst, guessing should fail altogether, with the function returning None. However, under any circumstances, it shouldn't raise an exception or return an incorrect class.

TL;DR

The final version of our function successfully overcomes most simple cases, and a few pitfalls as well.

In a nutshell, its implementation differentiates between bound methods and “unbound methods“ (functions) since in Python 3 there is no reliable way to extract the enclosing class from an “unbound method".

Several useful comments prompted additional changes, as detailed in the edits section below, producing the following improvements:

  • Limited handling for methods defined via descriptors, that aren't classified as ordinary methods or functions (for example, set.union, int.__add__ and int().__add__) and for built-in methods (for example set().union and io.BytesIO().__enter__).
  • Handling of functools.partial objects.

The resulting function is:

def get_class_that_defined_method(meth):
    if isinstance(meth, functools.partial):
        return get_class_that_defined_method(meth.func)
    if inspect.ismethod(meth) or (inspect.isbuiltin(meth) and getattr(meth, '__self__', None) is not None and getattr(meth.__self__, '__class__', None)):
        for cls in inspect.getmro(meth.__self__.__class__):
            if meth.__name__ in cls.__dict__:
                return cls
        meth = getattr(meth, '__func__', meth)  # fallback to __qualname__ parsing
    if inspect.isfunction(meth):
        cls = getattr(inspect.getmodule(meth),
                      meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0],
                      None)
        if isinstance(cls, type):
            return cls
    return getattr(meth, '__objclass__', None)  # handle special descriptor objects

A small request

If you decide to use this implementation, and encounter any caveats, please comment and describe what happened.


The Full Version

“Unbound methods” are regular functions

First of all, it's worth noting the following change made in Python 3 (see Guido's motivation here):

The concept of “unbound methods” has been removed from the language. When referencing a method as a class attribute, you now get a plain function object.

This makes it practically impossible to reliably extract the class in which a certain “unbound method“ was defined unless it's bound to an object of that class (or of one of its subclasses).

Handling bound methods

So, let us first handle the “easier case“ in which we have a bound method. Note that the bound method must be written in Python, as described in inspect.ismethod's documentation.

def get_class_that_defined_method(meth):
    # meth must be a bound method
    if inspect.ismethod(meth):
        for cls in inspect.getmro(meth.__self__.__class__):
            if meth.__name__ in cls.__dict__:
                return cls
    return None  # not required since None would have been implicitly returned anyway

However, this solution is not perfect and has its perils, as methods can be assigned in runtime, rendering their name possibly different than that of the attribute that they are assigned to (see example below). This problem exists also in Python 2. A possible workaround would be to iterate over all of the class's attributes, looking for one whose identity is that of the specified method.

Handling “unbound methods“

Now that we got that out of the way, we can suggest a hack that tries to handle “unbound methods”. The hack, its rationale, and some discouragement words can be found in this answer. It relies on manually parsing the __qualname__ attribute, available only from Python 3.3, is highly unrecommended, but should work for simple cases:

def get_class_that_defined_method(meth):
    if inspect.isfunction(meth):
        return getattr(inspect.getmodule(meth),
                       meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0],
                       None)
    return None  # not required since None would have been implicitly returned anyway

Combining both approaches

Since inspect.isfunction and inspect.ismethod are mutually exclusive, combining both approaches into a single solution gives us the following (with added logging facilities for the upcoming examples):

def get_class_that_defined_method(meth):
    if inspect.ismethod(meth):
        print('this is a method')
        for cls in inspect.getmro(meth.__self__.__class__):
            if meth.__name__ in cls.__dict__:
                return cls
    if inspect.isfunction(meth):
        print('this is a function')
        return getattr(inspect.getmodule(meth),
                       meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0],
                       None)
    print('this is neither a function nor a method')
    return None  # not required since None would have been implicitly returned anyway

Execution example

>>> class A:
...     def a(self): pass
... 
>>> class B:
...     def b(self): pass
... 
>>> class C(A, B):
...     def a(self): pass
... 
>>> A.a
<function A.a at 0x7f13b58dfc80>
>>> get_class_that_defined_method(A.a)
this is a function
<class '__main__.A'>
>>>
>>> A().a
<bound method A.a of <__main__.A object at 0x7f13b58ca9e8>>
>>> get_class_that_defined_method(A().a)
this is a method
<class '__main__.A'>
>>>
>>> C.a
<function C.a at 0x7f13b58dfea0>
>>> get_class_that_defined_method(C.a)
this is a function
<class '__main__.C'>
>>>
>>> C().a
<bound method C.a of <__main__.C object at 0x7f13b58ca9e8>>
>>> get_class_that_defined_method(C().a)
this is a method
<class '__main__.C'>
>>>
>>> C.b
<function B.b at 0x7f13b58dfe18>
>>> get_class_that_defined_method(C.b)
this is a function
<class '__main__.B'>
>>>
>>> C().b
<bound method C.b of <__main__.C object at 0x7f13b58ca9e8>>
>>> get_class_that_defined_method(C().b)
this is a method
<class '__main__.B'>

So far, so good, but...

>>> def x(self): pass
... 
>>> class Z:
...     y = x
...     z = (lambda: lambda: 1)()  # this returns the inner function
...     @classmethod
...     def class_meth(cls): pass
...     @staticmethod
...     def static_meth(): pass
...
>>> x
<function x at 0x7f13b58dfa60>
>>> get_class_that_defined_method(x)
this is a function
<function x at 0x7f13b58dfa60>
>>>
>>> Z.y
<function x at 0x7f13b58dfa60>
>>> get_class_that_defined_method(Z.y)
this is a function
<function x at 0x7f13b58dfa60>
>>>
>>> Z().y
<bound method Z.x of <__main__.Z object at 0x7f13b58ca9e8>>
>>> get_class_that_defined_method(Z().y)
this is a method
this is neither a function nor a method
>>>
>>> Z.z
<function Z.<lambda>.<locals>.<lambda> at 0x7f13b58d40d0>
>>> get_class_that_defined_method(Z.z)
this is a function
<class '__main__.Z'>
>>>
>>> Z().z
<bound method Z.<lambda> of <__main__.Z object at 0x7f13b58ca9e8>>
>>> get_class_that_defined_method(Z().z)
this is a method
this is neither a function nor a method
>>>
>>> Z.class_meth
<bound method type.class_meth of <class '__main__.Z'>>
>>> get_class_that_defined_method(Z.class_meth)
this is a method
this is neither a function nor a method
>>>
>>> Z().class_meth
<bound method type.class_meth of <class '__main__.Z'>>
>>> get_class_that_defined_method(Z().class_meth)
this is a method
this is neither a function nor a method
>>>
>>> Z.static_meth
<function Z.static_meth at 0x7f13b58d4158>
>>> get_class_that_defined_method(Z.static_meth)
this is a function
<class '__main__.Z'>
>>>
>>> Z().static_meth
<function Z.static_meth at 0x7f13b58d4158>
>>> get_class_that_defined_method(Z().static_meth)
this is a function
<class '__main__.Z'>

Final touches

  • The outcome generated by x and Z.y can be partially fixed (to return None) by verifying that the returned value is a class, before actually returning it.

  • The outcome generated by Z().z can be fixed by falling back to parsing the function's __qualname__ attribute (the function can be extracted via meth.__func__).

  • The outcome generated by Z.class_meth and Z().class_meth is incorrect because accessing a class method always returns a bound method, whose __self__ attribute is the class itself, rather than its object. Thus, further accessing the __class__ attribute on top of that __self__ attribute doesn't work as expected:

    >>> Z().class_meth
    <bound method type.class_meth of <class '__main__.Z'>>
    >>> Z().class_meth.__self__
    <class '__main__.Z'>
    >>> Z().class_meth.__self__.__class__
    <class 'type'>
    

    This can be fixed by checking whether the method's __self__ attribute returns an instance of type. However, this might be confusing when our function is invoked against methods of a metaclass, so we'll leave it as is for now.

Here is the final version:

def get_class_that_defined_method(meth):
    if inspect.ismethod(meth):
        for cls in inspect.getmro(meth.__self__.__class__):
            if meth.__name__ in cls.__dict__:
                return cls
        meth = meth.__func__  # fallback to __qualname__ parsing
    if inspect.isfunction(meth):
        cls = getattr(inspect.getmodule(meth),
                      meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0],
                      None)
        if isinstance(cls, type):
            return cls
    return None  # not required since None would have been implicitly returned anyway

Surprisingly, this also fixes the outcome of Z.class_meth and Z().class_meth which now correctly return Z. This is because the __func__ attribute of a class method returns a regular function whose __qualname__ attribute may be parsed:

>>> Z().class_meth.__func__
<function Z.class_meth at 0x7f13b58d4048>
>>> Z().class_meth.__func__.__qualname__
'Z.class_meth'

EDIT:

As per the issue raised by Bryce, it's possible to handle method_descriptor objects, like set.union, and wrapper_descriptor objects, like int.__add__, merely by returning their __objclass__ attribute (introduced by PEP-252), if such exists:

if inspect.ismethoddescriptor(meth):
    return getattr(meth, '__objclass__', None)

However, inspect.ismethoddescriptor returns False for the respective instance method objects, i.e. for set().union and for int().__add__:

  • Since int().__add__.__objclass__ returns int, the above if clause may be relinquished in order to solve the problem for int().__add__. Unfortunately, this doesn't address the matter of set().union, for which no __objclass__ attribute is defined. In order to avoid an AttributeError exception in such a case, the __objclass__ attribute isn't accessed directly, but rather via the getattr function.

EDIT:

As per the issue raised by x-yuri, it seems that our function fails to handle the method io.BytesIO().__enter__ since inspect doesn't identify it as a method, but rather as a built-in:

>>> inspect.ismethod(io.BytesIO().__enter__)
False
>>> inspect.isbuiltin(io.BytesIO().__enter__)
True

This is the same issue encountered above in regard to set().union:

>>> inspect.ismethod(set().union)
False
>>> inspect.isbuiltin(set().union)
True

Other than this peculiarity, we can handle such methods as ordinary methods and extract the defining class by traversing the MRO.

However, just to be on the safe side, we shall include an extra layer of protection and verify that the __self__ attribute of such methods, if defined, isn't None and that the __class__ attribute of that __self__ object, if defined, isn't None as well:

if inspect.ismethod(meth) or (inspect.isbuiltin(meth) and getattr(meth, '__self__', None) and getattr(meth.__self__, '__class__', None)):
    # ordinary method handling

Alas, this simple test fails for set().union because bool(set().union.__self__) evaluates to False since set().union.__self__ returns the empty set. Thus, an explicit test against None is required, producing the following fix:

if inspect.ismethod(meth) or (inspect.isbuiltin(meth) and getattr(meth, '__self__', None) is not None and getattr(meth.__self__, '__class__', None)):
    # ordinary method handling

A minor additional patch is advised in order to avoid a possible AttributeError exception when accessing the __func__ attribute during fallback to __qualname__ parsing. This is required since while the __func__ attribute is guaranteed to exist for an ordinary method, it's not necessarily defined for one of the type builtin_function_or_method, such as io.BytesIO().__enter__ and set().union.

def get_class_that_defined_method(meth):
    if inspect.ismethod(meth) or (inspect.isbuiltin(meth) and getattr(meth, '__self__', None) is not None and getattr(meth.__self__, '__class__', None)):
        for cls in inspect.getmro(meth.__self__.__class__):
            if meth.__name__ in cls.__dict__:
                return cls
        meth = getattr(meth, '__func__', meth)  # fallback to __qualname__ parsing
    if inspect.isfunction(meth):
        cls = getattr(inspect.getmodule(meth),
                      meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0],
                      None)
        if isinstance(cls, type):
            return cls
    return getattr(meth, '__objclass__', None)  # handle special descriptor objects

EDIT:

As per the suggestion put forward by user1956611, it's possible to handle partial objects by introducing a recursive call to seek out the original callable with which the partial object was created:

if isinstance(meth, functools.partial):
    return get_class_that_defined_method(meth.func)
Maurili answered 21/9, 2014 at 13:10 Comment(17)
This identifies @staticmethod's as methods with this base class. They really aren't. Once you have the class you should check that not isinstance( cls.__dict__[meth.__name__], staticmethod ). It is ambiguous whether you should also check for classmethod. (The point of using dict is to dodge substitutions done by get) . I don't think you need to scan up the mro in this case, because the definition will really lie in whatever class qualname indicates, but I am not entirely certain. (This insight comes from the new 'inspect.getattr_static' method implementation)Gallup
Also, re: the edit. set().union is bound. It has a self. So you can use the type of that as the declaring class. And set.union has an objclass.Gallup
@jobermark, thanks for your comments, but I don't quite follow. Can you provide an example for which the returned class is incorrect for a @staticmethod or a @classmethod? Also, regarding set().union and int().__add__, I'd rather refrain from trying to obtain its "self object" unless I'm certain that I'm dealing with a bound method.Maurili
The function takes an unbound method... set().__add__ is a bound method. And set.__add__ now has an objclass in Python 3.7. So this special case from the comment has disappeared.Gallup
Maybe I misunderstood. I guess I am after something other than the defining class. As named this is correct, but it is less useful than it might be. The primary use is going to be to check that I can take an object of the returned class, .__get__ the method onto it to bind, and use the result as expected. For staticmethods and classmethods this is not quite right. You should not bind it to anything if it is a staticmethod, and if it is a classmethod you should bind it to the class.Gallup
Your method doesn't work for _io.BytesIO.__enter__, as opposed to this solution. More on it here.Persimmon
@x-yuri, thanks for your input. I've adapted my answer to handle built-ins as well.Maurili
I had a need to inspect functions that might be wrapped as a partial function. Adding a check and recursive call at the top allows handling of any depth of partial function wrapping. I don't understand why I can't create code blocks. Good luck deciphering this mess. ~~~ def get_class_that_defined_method(meth: Callable): if isinstance(meth, functools.partial): return get_class_that_defined_method(meth.func) ~~~Jabot
@user1956611, thanks for your suggestion. I've incorporated it into my answer. On a side note, code formatting in comments is done using ` (the backtick symbol), as detailed here.Maurili
awesome post! do you have an example of the __func__ not being present in a builtin? i cant find anyCosentino
ah it seems like class Z(object): z = lambda: lambda: 1; get_class_that_defined_method(Z().z) is such a thing , i thinkCosentino
@studioj, I was actually referring to io.BytesIO().__enter__ and set().union which are of type builtin_function_or_method and don't have the attribute __func__. Thank you for asking this question, prompting me to clarify this point in my answer.Maurili
Wow. Glad I googled this before trying to figure it out myself, lol. :-)Ilocano
I had an issue when using this snippet with pytest --import-mode=importlib: https://mcmap.net/q/183333/-get-defining-class-of-method-object-when-using-pytest/2761174Weatherby
I didn't experience any issues with Python 3.8.10, pytest-7.0.0, pluggy-1.0.0. We can continue the discussion in the comments section of your question.Maurili
Is there any reason for using getattr(obj, attr, None) is not None over hasattr?Tiberius
The outcome is different if obj.attr is None. To elaborate: if obj.attr is None, then getattr(obj, attr, None) is not None evaluates to False while hasattr(obj, attr) evaluates to True.Maurili
S
47

The point you appear to be missing is, in Python 3 the "unbound method" type has entirely disappeared -- a method, until and unless it's bound, is just a function, without the weird "type-checking" unbound methods used to perform. This makes the language simpler!

To wit...:

>>> class X:
...   def Y(self): pass
... 
>>> type(X.Y)
<class 'function'>

and voila -- one less subtle concept and distinction to worry about. Such simplifications are the core advantage of Python 3 wrt Python 2, which (over the years) had been accumulating so many subtleties that it was in danger (if features kept being added to it) of really losing its status as a simple language. With Python 3, simplicity is back!-)

Strained answered 28/8, 2010 at 3:20 Comment(6)
That makes sense. I didn't even realize there was extra functionality for methods like the first-argument type checking--I just thought they would be normal functions with a few extra attributes. With that, I can completely understand the removal.Personally
@Tim, yes, the check that the first argument was of an instance of the .im_class (including subclasses thereof of course) is why im_class was kept in the first place (and the same layer of indirectness wrapped around functions to make unbound methods, as needs to be wrapped anyway to make bound ones). The underlying function always was (and still is for bound methods) the .im_func, BTW, never the method object itself.Strained
"With Python 3, simplicity is back!" sounds like it would make a good bumper sticker (or laptop sticker).Cidevant
Would anyone mind explaining how this answers tim's question? Thanks!Maxilliped
@GershomMaes, if a question was, "how do I get the defining class of a unicorn", are you really unable to see how "unicorns don't exist" answers it? The question being "how do I get the defining class of an unbound method in Python 3", clearly "unbound methods in Python 3 do not exist" answers it in exactly the same way: obviously, you can't "get the defining class" (or any other characteristic) of something that just does not exist!-)Strained
Ahh I see, I didn't realize the question was particularly asking about unbound methods! ThanksMaxilliped
A
9

Since python 3.6 you could accomplish what you are describing using a decorator that defines a __set_name__ method. The documentation states that object.__set_name__ is called when the class is being created.

Here is an example that decorates a method "in order to register it in a list of methods that serve a particular purpose":

>>> class particular_purpose: 
...     def __init__(self, fn): 
...         self.fn = fn 
...      
...     def __set_name__(self, owner, name): 
...         owner._particular_purpose.add(self.fn) 
...          
...         # then replace ourself with the original method 
...         setattr(owner, name, self.fn) 
...  
... class A: 
...     _particular_purpose = set() 
...  
...     @particular_purpose 
...     def hello(self): 
...         return "hello" 
...  
...     @particular_purpose 
...     def world(self): 
...         return "world" 
...  
>>> A._particular_purpose
{<function __main__.A.hello(self)>, <function __main__.A.world(self)>}
>>> a = A() 
>>> for fn in A._particular_purpose: 
...     print(fn(a)) 
...                                                                                                                                     
world
hello

Note that this question is very similar to Can a Python decorator of an instance method access the class? and therefore my answer as well to the answer I provided there.

Animalcule answered 22/1, 2019 at 21:46 Comment(1)
Be careful with this approach, as the decorator class (particular_purpose) will return itself and not the function, which means any decorators "above" it will incorrectly receive this object and likely malfunction. I would recommend using a class decorator (as shown here) if you are tagging and then post-processing functions in the class.Midland
M
2

A small extension for python 3.6 (python 2.7 worked fine) to the great answer of https://mcmap.net/q/182623/-get-defining-class-of-unbound-method-object-in-python-3

def get_class_that_defined_method(meth):
    if inspect.ismethod(meth):
        for cls in inspect.getmro(meth.__self__.__class__):
            if cls.__dict__.get(meth.__name__) is meth:
                return cls
        meth = meth.__func__  # fallback to __qualname__ parsing
    if inspect.isfunction(meth):
        class_name = meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0]
        try:
            cls = getattr(inspect.getmodule(meth), class_name)
        except AttributeError:
            cls = meth.__globals__.get(class_name)
        if isinstance(cls, type):
            return cls
    return None  # not required since None would have been implicitly returned anyway

I found the following adjustment was required for doctest

        except AttributeError:
            cls = meth.__globals__.get(class_name)

As for some reason, when using nose the inspect.getmodule(meth) didn't contain the defining class

Muddlehead answered 8/2, 2019 at 16:56 Comment(0)
S
0

I made a basic wrapper object that wraps a function and includes info about the parent class and module of the function.

I have included the 'get_class_that_defined_method' in Yoel's answer as the way to get the class info.

Here is a link to the GitHub gist of the file.

Speos answered 19/2, 2022 at 5:22 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.