Using __new__ to override __init__ in subclass

Question

I'm interested in using the __new__ functionality to inject code into the __init__ function of subclasses. My understanding from the documentation is that python will call __init__ on the instance returned by __new__. However, my efforts to change the value of __init__ in the instance before returning it from __new__ don't seem to work.

class Parent(object):

    def __new__(cls, *args, **kwargs):
        new_object = super(Parent, cls).__new__(cls)
        user_init = new_object.__init__
        def __init__(self, *args, **kwargs):
            print("New __init__ called")
            user_init(self, *args, **kwargs)
            self.extra()
        print("Replacing __init__")
        setattr(new_object, '__init__', __init__)
        return new_object

    def extra(self):
        print("Extra called")

class Child(Parent):

    def __init__(self):
        print("Original __init__ called")
        super(Child, self).__init__()

c = Child()

The above code prints:

Replacing __init__
Original __init__ called

but I would expect it to print

Replacing __init__
New __init__ called
Original __init__ called
Extra called

Why not?

I feel like Python is calling the original value of __init__, regardless of what I set it to in __new__. Running introspection on c.__init__ shows that the new version is in place, but it hasn't been called as part of the object creation.

Unconstitutional

What's your question? – Culbert

Answer 1

Well, the new object is expected to be empty before the __init__ is called. So probably python, as optimization, does not bother to query the object and goes to fetch __init__ straight from the class.

Therefore you'll have to modify __init__ of the subclasses themselves. Fortunately Python has a tool for that, metaclasses.

In Python 2, you set metaclass by setting special member:

class Parent(object):
    __metaclass__ = Meta
    ...

See Python2 documentation

In Python 3, you set metaclass via keyword attribute in the parent list, so

class Parent(metaclass=Meta):
    ...

See Python3 documentation

The metaclass is a base class for the class instance. It has to be derived from type and in it's __new__ it can modify the class being created (I believe the __init__ should be called too, but the examples override __new__, so I'll go with it). The __new__ will be similar to what you have:

class Meta(type):
    def __new__(mcs, name, bases, namespace, **kwargs):
        new_cls = super(Meta, mcs).__new__(mcs, name, bases, namespace, **kwargs)
        user_init = new_cls.__init__
        def __init__(self, *args, **kwargs):
            print("New __init__ called")
            user_init(self, *args, **kwargs)
            self.extra()
        print("Replacing __init__")
        setattr(new_cls, '__init__', __init__)
        return new_cls

(using the Python 3 example, but the signature in Python 2 seems to be the same except there are no **kwargs, but adding them shouldn't hurt; I didn't test it).

Milkfish

Answer 2

I suspect the answer is that __init__ is a special function, internally it is defined as a class method, and as a result cannot be replaced by reassigning it in an instance of the object.

In Python, all objects are represented by the PyObject in C, which has a pointer to a PyTypeObject. This contains a member called tp_init that I believe contains a pointer to the __init__ function.

The other solution works, because we are modifying the class, not an instance of the object.

Deception

PyTypeObject represents type object, not all objects, no? – Milkfish

Argh. Was looking for PyObject in the source code via github and found PyTypeObject instead. A PyObject has a ob_type method that does point to a PyTypeObject so I think my reasoning still could stands. It really depends on how the __init__ method is called. – Deception

Rewrote my answer to try and simplify. – Deception