Python 3.x: How should one override inherited properties from parent classes?

Question

A simple example probably shows more:

class NaturalNumber():
    def __init__(self, val):
        self._value = val

    def _get_value(self):
        return self._value

    def _set_value(self, val):
        if val < 0:
            raise ValueError(
                f"Cannot set value to {val}: Natural numbers are not negative."
            )
        self._value = val

    value = property(_get_value, _set_value, None, None)


class EvenNaturalNumber(NaturalNumber):
    # This seems superfluous but is required to define the property.
    def _get_value(self):
        return super()._get_value()

    def _set_value(self, val):
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )
        super()._set_value(val)

    # This seems superfluous but parent property defined with parent setter.
    value = property(_get_value, _set_value, None, None)

This is a simplified example from a real usecase where I want to inject code for testing over production classes. The principle is the same, and here I am introducing an extra validation to value in the EvenNaturalNumber class that inherits from the NaturalNumber class. My boss doesn't like me getting rid of all his decorators, so ideally a solution should work however the underlying class is written.

What would seem natural is:

class NaturalNumber():
    def __init__(self, val):
        self._value = val
    
    @property
    def value(self):
        return self._value
    
    @value.setter
    def value(self, val):
        if val < 0:
            raise ValueError(
                f"Cannot set value to {val}: Natural numbers are not negative."
            )
        self._value = val


class EvenNaturalNumber(NaturalNumber):
    @property
    def value(self):
        return super().value
    
    @value.setter
    def value(self, val):
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )
        super().value = val

But this errors with valid sets on EvenNaturalNumber.value (say, = 2). With AttributeError: 'super' object has no attribute 'value'.

Personally, I would say that's a bug in the python language! But I'm suspecting I have missed something.

---

I have found a solution using decorators but this seems rather convoluted:

class NaturalNumber():
    def __init__(self, val):
        self._value = val

    @property
    def value(self):
        return self._value

    @value.setter
    def value(self, val):
        if val < 0:
            raise ValueError(
                f"Cannot set value to {val}: Natural numbers are not negative."
            )
        self._value = val


class EvenNaturalNumber(NaturalNumber):
    @property
    def value(self):
        return super().value
    
    @value.setter
    def value(self, val):
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )
        super(type(self), type(self)).value.fset(self, val)

And another way is:

class NaturalNumber():
    def __init__(self, val):
        self._value = val
    
    @property
    def value(self):
        return self._value
    
    @value.setter
    def value(self, val):
        if val < 0:
            raise ValueError(
                f"Cannot set value to {val}: Natural numbers are not negative."
            )
        self._value = val


class EvenNaturalNumber(NaturalNumber):
    # This seems superfluous but is required to define the property.
    def _set_value(self, val):
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )
        NaturalNumber.value.fset(self, val)

    # This seems superfluous as parent property defined with parent setter.
    value = NaturalNumber.value.setter(_set_value)

But this second solution seems rather unsatisfactory as it makes use of the knowledge that the value property is defined in the NaturalNumber class. I don't see any way to iterate over the EvenNaturalNumber.__mro__ unless I do this in EvenNaturalNumber._set_value(self, val), but that's the job of super(), isn't it?

Any improvement or suggestions will be gratefully received. Otherwise, my boss is just going to have to live with super(type(self), type(self)).value.fset(self, val)!

---

Added 01/01/2024.

Many thanks to ShadowRanger for pointing out the error with super(type(self), type(self)); I completely agree this should be super(__class__, type(self)) (mia culpa).

Why this has arisen is as follows. My parent classes would be better named Device (equivalent to NaturalNumber above). These talk to external devices over a serial connection (using pyserial). Various attributes on the external devices are exposed in the class as properties. So a getter will write to the external device to query a value and return the reply as the value appropriately decoded and typed for the callee. The setter writes to the device to set the value there and then checks there has been an appropriate reponse from the device (all with apropriate error handling for exceptional circumstances). Needless to say, the whole system is about getting these devices to interact in interesting ways. All of this seems quite natural.

I am building a test system over the top of this so that (to some extent) the entire system can be tested (unit / functional / regression etc.) without actually being connected to any of the devices: I hesitate to use the phrase "test harness" but perhaps justified in this case. So the idea is to use inheritance over each Device class and have a MockDevice class that inherits from its parent and exposes the same properties. But each MockDevice class is initialised with a MockSerialConnection (as opposed to a real SerialConnection which the Device classes are initialised with), so that we can inject the expected responses into the MockSerialConnection, which are then read by the Device code and (hopefully) interpreted correctly, thus providing a mechanism to test changes to the code as the system develops. It is all smoke and mirrors, but hopefully in a good way.

For the MockDevice properties the getters and setters need to set the relevant serial communication and then call the relevant getters and setters of the parent Device class, so that we have good code coverage. There is quite a lot of multiple inheritance going on here too (thank goodness for the __mro__), and exactly where in the inheritance hierachy a method or property is defined isn't completely fixed (and may vary in the future). We start with AbstractDevice classes that essentially define functionality of the device (with a whole class heirachy here for similar devices with more or fewer features or functionality), the actual Device class represents a specific device (down to catalogue number) from a given manufacturer, which is not only dependant of the AbstractDevice but the communication protocol (and type of serial connection) together with specifics (such as the commands to send for a specific attribute).

The helper function solution while good for providing functionality that is easily overriden (library authors take note) doesn't quite fit the bill here for the purpose of testing. There is nothing stopping another developer down the road applying a "quick fix" to the setters and getters (rather than the relevant helper functions), that would never be picked up by the test system. I also don't really see the difference from my first code sample where I specifically define _get_value and _set_value and declare the property with value = property(_get_value, _set_value, None, None) and avoid the decorators.

I am also a little disatisfied with the @NaturalNumber.value.setter solution because it needs apriori knowledge that the property being set resides in the NaturalNumber class (the improvement over my earlier solution accepted). Yes, I can work it out how it is now but as a test system it should work if down the line a developer moves the functionality to another place in the inheritance hierarchy. The test system will still error but it means we will need to maintain code in two places (production and test); it seems preferable that the test system walks the mro to find the relevant class property to override. If one could @super().value.setter then that would be perfect, but one can't.

Notice that there is no problem in the getters:

class EvenNaturalNumber(NaturalNumber):
    @property
    def value(self):
        return super().value

works just fine. This is why I am tempted to think of this as a "language bug". The fact that super(__class__, type(self)).value.fset(self, val) works suggests (to me) that the python compiler would be able to detect super().value = val as a setter and act accordingly. I would be interested to know what others thought about this as a Python Enhancement Proposal (PEP), and if supportive what else should be addressed around the area of properties. I hope that helps to give a fuller picture.

Also if there are other suggestions on a different way to approach the test harness, that will also be gratefully received.

Dose

Not providing and answer, just an explanation. Your first example works because it creates a new property object in the subclass, while the second example fails because it tries to modify the superclass’s property object, which isn’t accessible in the subclass. Otherwise @shadowranger reply is probably the way to go, see #51880305 – Marci

Thank you culicidae, I hadn't found that question on stackoverflow, it does seem to be the same, but the answer here from ShadowRanger seems more complete; namely super(__class__, type(self)).value.fset(self, val). That seems to be the way I'll go (at least, for now). – Dose

I think your actual problem is that "For the MockDevice properties the getters and setters need to set the relevant serial communication and then call the relevant getters and setters of the parent Device class, so that we have good code coverage.". Either you should mock the getters and setters to always return a MockSerialConnection (and not test the parent getters/setters "for coverage" - if they contain relevant code that needs to be tested, do that in a separate test) or just use the original getter/setter code to assign the MockSerialConnection, and get coverage on them – Enfeeble

I think it is bad to use getters and setters to sync values with the device. There are several reasons for this but the biggest might be that getters and setters are not async and from my experience you will soon reach the point where you want to communicate with your devices in an async manner (using asyncio for example). – Lingo

Hi DarkMath, yes where we imploy getters and setters the code is not async: because we want to block until we have the response. Other devices are async but employ the same Mock class structure for the test system (essentially using a fifo queue as a back log of messages to be read). It is probably not worth diving into specifics, and I would say that overriding a property (getters, setters) is as natural as overriding any other function (my opinion). You mention there may be other reasons, I wil be grateful if you ellaborate. Thank you for your continued considerations. – Dose

Answer 1

So first off:

super(type(self), type(self)) is straight up wrong; never do that (it seems like it works when there is one layer, it fails with infinite recursion if you make a child class and try to invoke the setter on it).

Sadly, this is a case where there is no elegant solution. The closest you can get, in the general case, is the safe version of super(type(self), type(self)), using that code as-is, but replacing super(type(self), type(self)) with super(__class__, type(self)). You can simplify the code a little by having the child class use a decorator rather than manually invoking .setter, getting the benefits of your value = NaturalNumber.value.setter(_set_value) solution more succinctly:

class EvenNaturalNumber(NaturalNumber):
    # No need to redefine the getter, since it works as is
    # Just use NaturalNumber's value for the decorator
    @NaturalNumber.value.setter
    def value(self, val):
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )
        # Use __class__, not type(self) for first argument,
        # to avoid infinite recursion if this class is subclassed
        super(__class__, type(self)).value.fset(self, val)

__class__ is what no-arg super() uses to get the definition-time class for the method (whenever super or __class__ are referenced in a function being defined within a class, it's given a faked closure-score that provides __class__ as the class the function was defined in). This avoids the infinite recursion problem (and is slightly more efficient as a side-benefit, since loading __class__ from closure scope is cheaper than calling type(self) a second time).

That said, in this particular case, the best solution is probably to do as DarkMath suggests, and use an internal validation method that the setter can depend on, so the property need not be overridden in the child at all. It's not a general solution, since the changes aren't always so easily factored out, but it's the best solution for this particular case.

Dotard

Hi ShadowRanger, thank you for your considered response. Especially, for pointing out the problem with super(type(self), type(self)). I absolutely agree now this should be super(__class__, type(self)) for the purpose of inheritance from my child class (mia culpa). I will give a fuller reply to get a little deeper into the issue and why it has arisen. – Dose

Answer 2

I'm not sure why you want to call a parent property in a overridden property. Can you please explain why you want to do that?

I mean, the normal my of doing this would be a helper method that can be overridden and called easily:

class NaturalNumber:
    def __init__(self, val):
        self._value = val

    @property
    def value(self):
        return self._value

    @value.setter
    def value(self, val):
        self._validate(val)
        self._value = val

    def _validate(self, val: int | float) -> None:
        if val < 0:
            raise ValueError(
                f"Cannot set value to {val}: Natural numbers are not negative."
            )


class EvenNaturalNumber(NaturalNumber):
    def _validate(self, val: int | float) -> None:
        super()._validate(val)
        if val % 2:
            raise ValueError(
                f"Cannot set value to {val}: Even numbers are divisible by 2."
            )

Another possible way would be the avoid getter and settes completly and use frozen dataclasses instead with a copy_with() method. This also has the nice benefit that you get rid of possible side-effects due to the setters. Here's the code example:

from dataclasses import dataclass, fields
from typing import override, Self


@dataclass(frozen=True)
class NaturalNumber:
    value: int | float
    name: str  # just for demonstration purposes of copy_with()

    def copy_with(self, **kwargs) -> Self:
        """ Generic implementation that independent of concrete attributes. """
        current_values = {field.name: getattr(self, field.name) for field in fields(self)}
        return self.__class__(**(current_values | kwargs))

    def __post_init__(self) -> None:
        if self.value < 0:
            raise ValueError(
                f"Cannot set value to {self.value}: Natural numbers are not negative."
            )


@dataclass(frozen=True)
class EvenNaturalNumber(NaturalNumber):
    @override
    def __post_init__(self) -> None:
        super().__post_init__()

        if self.value % 2:
            raise ValueError(
                f"Cannot set value to {self.value}: Even numbers are divisible by 2."
            )


if __name__ == '__main__':
    number = EvenNaturalNumber(value=42, name='foo')
    number = number.copy_with(value=2)

Lingo

In the dataclasses example, copy_with is reinventing a wheel; dataclasses already provides dataclasses.replace, so, if you must have a method to do it, the implementation would just be return dataclasses.replace(self, **kwargs). It guarantees __post_init__ is still called, so the validation will still work. You can just skip providing copy_with and leave it to your users to call dataclasses.replace themselves directly of course; obj.copy_with(a=b) is not improving on dataclasses.replace(obj, a=b). – Dotard

Thank you very much! I didn't know that dataclass.replace exists, what a shame. – Lingo

Thank you DarkMath for your reply. I have added to my original post a much fuller description of the use case (which is to create a test harness which makes the use of helper functions not quite so good). I am not sure where you are going with the @dataclass example, but if you suggest this is a way froward for the test system I describe then I will look further into that. – Dose