how does metaclass work with the MRO list when super() is called?

Question

I'm really confused by the following code sample:

class Meta_1(type):
    def __call__(cls, *a, **kw):             # line 1
        print("entering Meta_1.__call__()")  

        print(cls)                           # line 4
        print(cls.mro())                     # line 5
        print(super(Meta_1, cls).__self__)   # line 6

        rv = super(Meta_1, cls).__call__(*a, **kw)  # line 7
        print("exiting Meta_1.__call__()")
        return rv


class Car(object, metaclass=Meta_1):

    def __new__(cls, *a, **kw):
        print("Car.__new__()")
        rv = super(Car, cls).__new__(cls, *a, **kw)
        return rv

    def __init__(self, *a, **kw):
        print("Car.__init__()")
        super(Car,self).__init__(*a, **kw)

if __name__ == '__main__':

    c = Car()

The print message for this code is:

entering Meta_1.__call__()
<class '__main__.Car'>                      # line 4
[<class '__main__.Car'>, <class 'object'>]  # line 5
<class '__main__.Car'>                      # line 6
Car.__new__()
Car.__init__()
exiting Meta_1.__call__()

The result shows that cls of line 4 is the Car class and its MRO list is:
[<class '__main__.Car'>, <class 'object'>]

However, line 6 shows that super(Meta_1, cls).__self__ is also the Car class.

I am really confused that:

1. In line 7, It seems that super(Meta_1, cls).__call__(*a, **kw) eventually lead to type.__call__. But, to my knowledge, super(arg1, arg2) will look into the MRO of the second input argument to find the first input argument, and return the next class to it. But in line 6 and 7 of my code, the MRO for 2nd argument(Car), does not contain the 1st input argument(Meta_1), you cannot find Meta_1 in the MRO for Car. so why would super(Meta_1, cos) take us to invoke type.__call__ ??

2. if super(Meta_1, cls).__self__ is the Car class, then line 7 means it's Car's __call__ that's being called? But calling the Car class took us to line 1 in the first place, right? wouldn't that be a loop?

Drayton

__call__ is an instance method of Meta_1, and Car is an instance of Meta_1. That means Car() is really short for Meta_1.__call__(Car). – Obstreperous

Meta_1 is not in the MRO of cls for the same reason that type is not in the MRO of an "ordinary" class. – Obstreperous

@Obstreperous Thank you. Then why should we put Meta_1 as the first argument in super()? – Drayton

Because that call is part of the definition of Meta_1.__call__. It's the same reason you use super(Car, self).__init__ in Car.__init__. – Obstreperous

@Obstreperous I'm still confused. The definition of Meta_1.__call__ is line 1, right? how does that relates to super() function? – Drayton

The purpose of type.__call__ (which you are overriding in Meta_1) is to ensure that the class's __new__ method gets called, so that you can write Car() instead of Car.__new__(). Using super().__call__ ensures that type.__call__ (eventually) gets called rather than you having to invoke cls.__new__ yourself. – Obstreperous

@Obstreperous so super(Meta_1, cls).__call__(*a, **kw) in line 7 basically invokes type.__call__, which then executes Car's __new__, is that accurate? This makes very good sense. I just don't see how super(Meta_1, cls) leads us to type. – Drayton

Because type is in the MRO of Meta_1. – Obstreperous

@Obstreperous I understand type is in the MRO for Meta_1, but in my code above, we are dealing with the MRO for cos (which is Car), and it does not contain Meta_1, right? – Drayton

@Obstreperous my question is simple: super() will look into the MRO of the second input argument to find the first input argument, and return the next class to it. But in my case, the MRO for 2nd argument, a.k.a Car, does not contain the 1st input argument, a.k.a. Meta_1, you cannot find Meta_1 in the MRO for Car. so why would super(Meta_1, cls) take us to type.__call__ ?? – Drayton

super(Meta_1, cls) does not use cls.mro(). Meta_1 is not in the MRO for Car, because Car does not inherit from Meta_1. – Obstreperous

@Obstreperous Isn't it true that super() will look into the MRO of the second input argument to find the first input argument, and return the next class to it? – Drayton

The mro method returns the MRO that is used for instances of the class. The wording is tricky, but cls.mro() returns the list of classes that get checked for instances of cls. – Obstreperous

@Obstreperous whose MRO does super(Meta_1, cls) use? Meta_1 's MRO? – Drayton

In theory; as far as I can tell, though, that's all special-cased in the implementation and not explicitly visible in Python itself. – Obstreperous

@Obstreperous So, usually super(A, B) would use B's MRO, but in my case in line 6 and 7, super(A, B) actually uses A's MRO. We don't know why. The logic behind it remains a mystery. oh that's frustrating. – Drayton

B doesn't have an MRO if it's not a type; type(B) does. There is no mystery here, only possibly confusion because cls (like any object) has a type, but also is a type. You need to be clear about what role cls is playing before you can think about which MRO is applicable. When cls is the second argument, you use the MRO determined by its type; when it is the first argument, you use the MRO determined by the type of the second argument (which is, very often, the first argument). – Obstreperous

Excellent answer here: #6967272 – Drayton

Answer 1

You are confusing a few concepts. The first of them is confusing the Metaclass with the class inheritance hierarchy.

Both things are ortogonal - looking at Car's mro will show you the inheritance tree for that class, and that does not include the metaclass. In other words, no Meta_1 should not, by any means, be in the MRO (or inheritance Tree).

The metaclass is the class' type - that is, it has the templates and methods to create the class object itself. As such, it has the "mechanisms" to build the class MRO itself, and to call the class' __new__ and __init__ (and __init_subclass__ and initialize the descriptors calling their __set_name__).

So, calling a class object, as calling any instance in Python will run the code in it's class __call__ method. In the case of a class, it happens that "calling" the class is the way to create a new instance - and what does that is the metaclass' __call__.

The other thing you are misunderstanding there is the super() object. Super() is not actually the superclass, neither an instance of the superclass - it is rather a proxy object, that will relay any attribute retrieval or method call to methods and attributes on the proper superclass. As part ot the mechanism super() uses to be able to act as a proxy, is to have the instance where it is called as its own __self__ attribute. In other words, the __self__ attribute is an ordinary attribute on the (proxy) object returned by super() call - it is picked from the second argument, or automatically in Python 3 - and it is used internally when the super object is used as a proxy to get act as if it were accessing attributes or methods on the "superclass" of that instance. (The instance annotated in __self__).

When you use super() inside the metaclass, the class proxied is the metaclass's superclass, which is type, not Car's superclass, object.

And so to yours second question:

2. if super(Meta_1, cls).__self__ is the Car class, then line 7 means it's Car's __call__ that's being called? But calling the Car class took us to line 1 in the first place, right? wouldn't that be a loop?

As said above, the super() call from the metaclass' __call__ will call type.__call__, and it will get the class Car as its cls parameter. That method in turn, will run Car.__new__ and Car.__init__ as the normal process to instantiate the class.

Rhine

Thank you for your answer. On this sentence you say about super: "....And one of the things it has to be able to do that is to anotate the instance where it is called from in its own __self__ attribute. ", I'm confused: do you mean that on line 6, the super's own __self__ attribute is annotating the instance where super is called? – Drayton

by the way: "...it is rather a proxy object, that will rely any attribute retrieval or method call to methods and attributes on the proper superclass.... ". Is it a typo to use the word "rely"? you meant "relay" ? sorry i'm not native english speaker – Drayton

I still don't understand why Meta_1 is passed to super() as the first argument. – Drayton

The first argument to super is almost always the class in which the method you are defining appears, so much so that in Python 3 that's the default value if you simply call super(), with no arguments. (There are reasons to use a different class in the MRO, but they are rare and you really shouldn't worry about it at this time.) – Obstreperous

The first argument to super serves as the starting point for the MRO search; super(A, obj) returns a proxy to the class after A in the MRO for object obj. – Obstreperous

As fr your question on the first comment, I will clarify the text in the answer itself – Rhine

@Obstreperous I understand but that is exactly why I'm confused: you said "super(A, obj) returns a proxy to the class after A in the MRO for object obj", but in my case, the MRO for object obj does not contain A. The MRO for Car does not contain Meta_1, you cannot find Meta_1 in the MRO for Car. Right? – Drayton

@Rhine Thank you for the edit. I get that super() is not the actually "dad" class but it is a proxy object. What I don't get is: super() will look into the MRO of the second input argument to find the first input argument, and return the next class to it. In my case, The MRO for 2nd argument, a.k.a Car, does not contain the 1st input argument, a.k.a. Meta_1, you cannot find Meta_1 in the MRO for Car. – Drayton

@Rhine quoting you: ".....When you use super() inside the metaclass, the class proxied is the metaclass's superclass, which is type ....". I get that this is what actually happened, but I don't see how: I thought calling super(A, obj) would return a proxy to the class after A in the MRO for obj. You are saying that if we call super(A, obj) inside the metaclass, it doesn't use the 1st and 2nd input arguments in the usual way, i.e. to proxy the class after A in the MRO for obj ? – Drayton

If you call super(A, ...) it will, yes, create a proxy for the next superclass of A on its MRO. And that is not Meta_1 or type, but object. It does not care where the call is done, if inside the metaclass or not. But as @Obstreperous puts it: one almost never needs to use super for a class in clde that is not inside the class that needs to use super itself. So much that it is the ONLY case where the Python language has special features to magically autofill the arguments in a function call. – Rhine

(Even the call to methods that insert self is done in code that is completely customizable in Python code. The implicit parameters to super() however, are created at compile time, and have no pure Python equivalent) – Rhine

@Thank you for your patience, first of all. Could you please answer specifically to this question: "super() will look into the MRO of the second input argument to find the first input argument, and return the next class to it. But in my case, the MRO for 2nd argument, a.k.a Car, does not contain the 1st input argument, a.k.a. Meta_1, you cannot find Meta_1 in the MRO for Car. so why would super(Meta_1, cos) take us to type.__call__ ?? " – Drayton

The second input argument to super must be an instance of the first argument. And it is the MRO of the first argument that is looked. And the MRO for Meta_1 is (Meta_1, type) . The MRO for Car is (Car, object). As it is in the text above, the class inheritance tree has nothing to do with the class inheritance tree. – Rhine

If you simply do not put an explicit argument to super, as is the best practice, it will just work. Only very, very specific code should ever need to use super outside a subclass. – Rhine

Answer 2

It's important to pay attention to what values are being used as each argument to super. The primary purpose of super is to perform attribute lookup according to some method-resolution order (MRO). The second argument determines which MRO to use; the first determines where to start looking.

An MRO is always defined by a class; when performing method resolution on an instance, we use the MRO of the class of which that instance is a type.

In the class

class Meta_1(type):
    def __call__(cls, *a, **kw):             # line 1
        print("entering Meta_1.__call__()")  

        print(cls)                           # line 4
        print(cls.mro())                     # line 5
        print(super(Meta_1, cls).__self__)   # line 6

        rv = super(Meta_1, cls).__call__(*a, **kw)  # line 7
        print("exiting Meta_1.__call__()")
        return rv

we see two uses of super. Both take the same arguments. cls is some object passed as the first argument to Meta_1.__call__. That means we'll use the MRO provided by type(cls), and we'll use the first class found after Meta_1 that provides the desired method. (In the first call, __self__ is an attribute of the proxy object itself, rather than an attribute or method of the class whose proxy super returns.)

When you run your code, you see that cls is bound to your Car type object. That's because Car() is implemented by type(Car).__call__(); since Car uses Meta_1 as its metaclass, type(Car) is Meta_1.

cls.mro() is irrelevant, because that's the MRO used by instances of cls.

The MRO of Meta_1 itself can be seen with

>>> Meta_1.mro(Meta_1)
[<class '__main__.Meta_1'>, <class 'type'>, <class 'object'>]

(mro is an instance method of the type class, and so requires the seemingly redundant instance of type as an argument. Keep in mind that cls.mro() is equivalent to type(cls).mro(cls).)

So line 7 is a call to type.__call__, in order to create an instance of cls that Meta_1.__call__ can return.

Obstreperous

This answer concludes all the comments and answers of this question. Well explained!! – Drayton

quick question: when you say "....cls is some object passed as the first argument to Meta_1.__call__. That means we'll use the MRO provided by type(cls) .....", here super() has not been called yet at line 1, so I don't see how putting cls as the first argument leads us to using the MRO of type(cls) instead of cls. Can you elaborate? – Drayton

Answer 3

This is an excellent answer from the original post by Michael Ekoka where my sample code came from: Using the __call__ method of a metaclass instead of __new__?

Basically, I need to get a better understanding of how super() works.

quote:

super will indeed use cls to find the MRO, but not the way one might think. I'm guessing you thought it would do something as direct as cls.__mro__ and find Meta_1. Not so, that's Class_1's MRO you're resolving by doing that, a different, unrelated MRO, and Meta_1 isn't a part of it (Class_1 does not inherit from Meta_1). cls even having an __mro__ property is just an accident due to it being a class. Instead, super will look up the class (a metaclass in our case) of cls, i.e. Meta_1, then will look up the MRO from there (i.e. Meta_1.__mro__).

Drayton