Given a method, how do I return the class it belongs to in Python 3.3 onward?

Question

Given x = C.f after:

class C:
    def f(self):
        pass

What do I call on x that will return C?

The best I could do is execing a parsed portion of x.__qualname__, which is ugly:

exec('d = ' + ".".join(x.__qualname__.split('.')[:-1]))

For a use case, imagine that I want a decorator that adds a super call to any method it's applied to. How can that decorator, which is only given the function object, get the class to super (the ??? below)?

def ensure_finished(iterator):
    try:
        next(iterator)
    except StopIteration:
        return
    else:
        raise RuntimeError

def derived_generator(method):
    def new_method(self, *args, **kwargs):
        x = method(self, *args, **kwargs)
        y = getattr(super(???, self), method.__name__)\
            (*args, **kwargs)

        for a, b in zip(x, y):
            assert a is None and b is None
            yield

        ensure_finished(x)
        ensure_finished(y)

    return new_method

Tragedienne

How about x.im_class? – Haifa

@dano: doesn't work in my ipython 3.4? – Tragedienne

possible duplicate of Get class that defined method – Vermination

Ah, sorry, it's python 2 only. – Haifa

@yoel: yes, but I've updated my question to specify python 3. I suspect it's not possible since Python 3 seems to have done away with types.MethodType for unbound methods? – Tragedienne

Is there are particular use-case you want this information for? In Python 3, the concept of unbound methods has been removed, so x is just a regular function that isn't directly tied to C at all. – Haifa

@dano: just asked that in my most recent question: #25922037 – Tragedienne

What about the answer in this SO question: #961548 . Probably uglier than what you have but would work on Python2 (but that isn't a requirement of yours) – Unclad

@MichaelPetch It uses the same x.im_class that isn't available in Python 3.x – Conquer

Sorry, you are correct. Accidentally ran on Python2.7 – Unclad

It's worth noting you shouldn't use im_class or self.__class__ as an argument to super (which is the OP's intention according to his other question), because if you ever subclass C you'll get infinite loops when f is called. – Haifa

Maybe something like this - x.__globals__['.'.join(x.__qualname__.split('.')[:-1])]? Still ugly, but at least without an exec... – Vermination

have you tried x.__self__ for the object instance? or x.__self__.__class__ for the class – Peruke

@Ashwin: AttributeError: 'function' object has no attribute '__self__' – Tragedienne

have you looked at this? #3589811 – Peruke

I think as long as you do x = C.f you may have issues. Since you are operating on the class itself (and not an instance of the class) the function x becomes detached. Only way I see would be to get the function on f through an instance like x = C().f – Unclad

@MichaelPetch: I am still interested to know if qualname is required to solve this problem, so this was not meant to be a Python 3.3 question, but it's fine as there is no reason for anyone to be using Python 3.0–3.2. – Tragedienne

It appears under 3.0 to 3.2 you may not have any alternatives. The only thing I suggest is that if you want to keep a method associated with its class (and have it seen as a method) after being assigned a variable - under 3.0-3.2 you may have to consider getting a method through an instance (and not the class). So rather than x = C.f you could used x = C().f. However if that doesn't suit your needs then I do not know of another solution at this time. – Unclad

@eryksun: No, C.f is a method: it is a function that belongs to the class C that takes a reference to self. It's true that Python 3 stopped using the MethodType for unbound methods, but semantically it is a method. – Tragedienne

@NeilG, it's semantically a method with respect to OOP design, but technically it's just a function object. Python lets you add a function (method) to a class at runtime, so the function's __qualname__ isn't necessarily related to the class. Also, parsing __qualname__ like this is beyond its intended usage. I found one issue with respect to closure <locals>; I'm not sure what other corner cases exist now or may come up in future releases. – Esemplastic

@eryksun: True. If a function is added to a class at runtime, how can a decorated version of it call super? I guess it would be impossible. It is unfortunate that Python does not have a __parent__ member for modules, classes, and class members so that the declaration structure can be walked. – Tragedienne

@eryksun: Unfortunately, I'm using the decorator on methods in the class and the class name is not defined until the class definition is complete. – Tragedienne

@NeilG, ok, maybe this will work. Use a decorator to set an identifying attribute on the target functions (e.g. func._derived = True). Then define derived_generator in a class decorator that creates a closure over the class, so you can call super(cls, self). Loop over the class dict to apply this to all of the _derived functions. – Esemplastic

@eryksun: Yes, great idea. It avoids my solution of using a metaclass, which I wanted to avoid. You should copy this comment into an answer to my other question. – Tragedienne

Since How do I call super in a method decorator in Python 3 is just the same question with a use case attached, and the answer to that question would be "do what's in this question", I closed it as a dup and copied the use case over. @NeilG, you may want to review the edit and undo if you think it's not appropriate. – Gillead

Answer 1

If your aim is to get rid of the exec statement, but are willing to use the __qualname__ attribute, even though you are still required to manually parse it, then at least for simple cases the following seems to work:

x.__globals__[x.__qualname__.rsplit('.', 1)[0]]

or:

getattr(inspect.getmodule(x), x.__qualname__.rsplit('.', 1)[0])

I'm not a Python expert, but I think the second solution is better, considering the following documentation excerpts:

• from What's new in Python 3.3:

  Functions and class objects have a new __qualname__ attribute representing the “path” from the module top-level to their definition. For global functions and classes, this is the same as __name__. For other functions and classes, it provides better information about where they were actually defined, and how they might be accessible from the global scope.

• from __qualname__'s description in PEP 3155:

  For nested classed, methods, and nested functions, the __qualname__ attribute contains a dotted path leading to the object from the module top-level.

EDIT:

1. As noted in the comments by @eryksun, parsing __qualname__ like this goes beyond its intended usage and is extremely fragile considering how __qualname__ reflects closures. A more robust approach needs to exclude closure namespaces of the form name.<locals>. For example:

   >>> class C:
   ...     f = (lambda x: lambda s: x)(1)
   ... 
   >>> x = C.f
   >>> x
   <function C.<lambda>.<locals>.<lambda> at 0x7f13b58df730>
   >>> x.__qualname__
   'C.<lambda>.<locals>.<lambda>'
   >>> getattr(inspect.getmodule(x), x.__qualname__.rsplit('.', 1)[0])
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   AttributeError: 'module' object has no attribute 'C.<lambda>.<locals>'
   

   This specific case can be handled in the following manner:

   >>> getattr(inspect.getmodule(x),
   ...         x.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0])
   <class '__main__.C'>
   

   Nonetheless, it's unclear what other corner cases exist now or may come up in future releases.

2. As noted in the comment by @MichaelPetch, this answer is relevant only for Python 3.3 onward, as only then the __qualname__ attribute was introduced into the language.

   • However, according to @WouterBolsterlee, github.com/wbolster/qualname provides an equivalent for older Python versions.

3. For a complete solution that handles bound methods as well, please refer to this answer.

Vermination

My concern here is that the target is Python 3. But __qualname__ didn't get added until Python 3.4. So does the question have to be amended to be Python 3.4 and higher? This solution will not work with Python 3.2 and 3.3 variants. – Unclad

@MichaelPetch, I wasn't aware of that. I've edited my answer to reflect your comment. Thanks! – Vermination

I've assumed it's safe to use __qualname__ since the OP used it in his question as a possible solution. I'll suggest a tag edit of the question to reflect your observation. – Vermination

Yeah, he used it in the question body. Also, it seems that it was introduced in Python 3.3. – Vermination

@MichaelPetch: Just FYI downvotes are for unclear or bad questions. It would have been nice to have a Python 3 solution, and an answer that says it's not possible in Python 3.0–3.2, but this is how you do it in Python 3.3+ is a good answer to my question. – Tragedienne

@MichaelPetch: Regardless of your downvote, I think you should consider upvoting Yoel's answer since you're paying such close attention to this question. – Tragedienne

It is a bad question because the title and tag do not match what the accepted answer was. If the accepted answer doesn't match the question - question is bad. – Unclad

@MichaelPetch: All he had to do was say that it wasn't possible in 3.0–3.2. – Tragedienne

@NeilG I don't think he knew. As he said he went off the fact that in one of your comments you were using __qualname__ therefore as he said he assumed he was okay to use it. I had to verify it worked on 3.3 and it does so I will remove my down vote on the question and upvote the answer. – Unclad

github.com/wbolster/qualname provides a qualname equivalent for older python versions. – Precincts

@WouterBolsterlee: Thanks, I've edited my answer accordingly. – Vermination

Answer 2

I'll contribute one more option that relies on the gc module to follow references backwards.

It relies on implementation details that certainly aren't guaranteed and certainly won't work on all Python implementations. Nevertheless, some applications may find this option preferable to working with __qualname__.

You actually need two hops backwards, because the class hides a dict inside it, which holds the member function:

def class_holding(fn):
    '''
    >>> class Foo:
    ...     def bar(self):
    ...         return 1
    >>> class_holding(Foo.bar)
    <class Foo>
    '''
    for possible_dict in gc.get_referrers(fn):
        if not isinstance(possible_dict, dict):
            continue
        for possible_class in gc.get_referrers(possible_dict):
            if getattr(possible_class, fn.__name__, None) is fn:
                return possible_class
    return None

Walford

This works on its own as in class_holding((Foo.bar), but I couldn't get it to work within a decorator, as the asker suggests (I'm looking for that too). gc.get_referrers(orig_method doesn't return any dicts, only <cell at 0x10bcefc10: function object at 0x10bcf1e40>>. – Smash