Workaround for lack of intersection types with Python generics?

Question

I'm hitting an issue that would be easily solved by intersection types (currently under discussion but not yet implemented) and was wondering what the cleanest workaround is.

Current setup and problem

My current setup roughly corresponds to the following ABC hierarchy for animals. There are a number of animal 'features' (CanFly, CanSwim, etc) defined as abstract subclasses (though they could also have been defined as mixins).

from abc import ABC, abstractmethod

class Animal(ABC):
    @abstractmethod
    def name(self) -> str: ...  
    
class CanFly(Animal):
    @abstractmethod
    def fly(self) -> None: ...
    
class CanSwim(Animal):
    @abstractmethod
    def swim(self) -> None: ...

With this I define specific classes of animals, both abstract and concrete:

class Bird(CanFly):
    def fly(self) -> None:
        print("flap wings")
    
class Penguin(Bird, CanSwim):
    def name(self) -> str:
        return "penguin"
    def swim(self) -> None:
        print("paddle flippers")

I also define a generic class for petting specific types of animals:

from typing import Generic, TypeVar

T = TypeVar("T", bound=Animal, contravariant=True)

class Petter(Generic[T], ABC):

    @abstractmethod
    def pet(self, a: T) -> None:
        ...

However, there is no way that I know of to specify a Petter for an intersection of features: e.g. for all animals that can both fly and swim.

class CanFlyAndSwim(CanFly, CanSwim):
    pass
        
class CanFlyAndSwimPetter(Petter[CanFlyAndSwim]):

    def pet(self, a: CanFlyAndSwim):
        a.name()
        a.fly()
        a.swim()
    
        
CanFlyAndSwimPetter().pet(Penguin())  # type error, as Penguin isn't a subclass of CanFlyAndSwim

I could try to work around this by insisting that Penguin explicitly inherit from CanFlyAndSwim, but this isn't scalable to more feature combinations.

Using Protocols instead?

Another approach I tried is to use Protocols instead:

from typing import Protocol

class AnimalProtocol(Protocol):
    def name(self) -> str: ...

class FlyProtocol(AnimalProtocol, Protocol):
    def fly(self) -> None: ...

class SwimProtocol(AnimalProtocol, Protocol):
    def swim(self) -> None: ...

With these we can indeed define a useful protocol intersection. After changing the type variable T upper bound to AnimalProtocol, we can write:

class FlyAndSwimProtocol(FlyProtocol, SwimProtocol, Protocol):
    ...

class FlyAndSwimProtocolPetter(Petter[FlyAndSwimProtocol]):

    def pet(self, a: FlyAndSwimProtocol):
        a.name()
        a.fly()
        a.swim()
    
        
FlyAndSwimProtocolPetter().pet(Penguin())  # ok

However, replacing the ABCs with Protocols removes the explicit class hierarchy when defining animals, which is useful both for documentation and for checking that all the relevant methods have been implemented. We could try keeping both the ABCs and the Protocols, though that involves significant code duplication, unless there's some way to define one from the other?

Is there a clean solution to all this?

Allargando

(BTW I do know that penguins can't fly. It was just the first example I could think of!) – Allargando

why do i feel i know what the use case is? ;) Have you tried composition over inheritance? I realise that's a very broad question that might not lead anywhere – Bonbon

"which is useful ... for checking that all the relevant methods have been implemented". Doesn't the type checker complain if you don't implement all the methods on the protocol then try to use it where the protocol is expected? – Bonbon

The type checker will complain if you try to use an animal with a petter if the animal doesn't implement all the methods required by the petter's protocol type. However, it won't complain if you just define an animal that's missing some methods: explicitly inheriting from the protocol doesn't check that you implement that protocol's methods (of which there may be many). – Allargando

The ABC approach gives a useful sanity check when defining the animals, not just when you happen to use them with something you expected to work. – Allargando

"explicitly inheriting from the protocol doesn't check that you implement that protocol's methods". that's news to me, and very unfortunate. It looks like it might be up to the type checker. It's not quite as sane as you're after, but mypy requires protocol methods are implemented to be able to instantiate the class if the protocol is explicitly subclassed and the method is marked @abstractmethod – Bonbon

That's actually great! (I think) I didn't know Protocols could behave like ABCs if you decorate the methods with \@abstractmethod. This way I can still use them for defining if I want to (though arbitrary structural subtyping is allowed when using them). And if I define the protocols as \@runtime_checkable I can check them at runtime too. – Allargando

Don't try to write Java in Python. – Bolin

Answer 1

First and foremost, remember that Python does duck typing extensively.

So the best solution in this case IMHO, is to:

• Design your data using inheritance, but
• Access attributes and methods using protocols

In other words, yeah I think you will have to mix the two approaches.

Intermolecular

Yes! Ad-hoc polymorphism usually plays nicer with Python's somewhat rigid typing constructs than explicitly designing class hierarchies. It may even make more sense to not design data using inheritance, depending on the exact details of OP's use case. – Konyn

Answer 2

It maybe offtop... But i think best way is to use composition over inheritance. It will eliminate hell of a lot of classes and be more simpler.

from abc import ABC, abstractmethod
from dataclasses import dataclass


class AnimalFeature(ABC):
    @abstractmethod
    def action(self) -> None:
        ...


class CanFly(AnimalFeature):
    @staticmethod
    def action() -> None:
        print("flap wings")


class CanSwim(AnimalFeature):
    @staticmethod
    def action() -> None:
        print("paddle flippers")


@dataclass
class Pet:
    name: str
    features: list[AnimalFeature] | None

    def pet(self):
        print(self.name)

        for feature in self.features:
            feature.action()


peter_pet = Pet(
    name="Peter",
    features=[CanFly, CanSwim],
)
peter_pet.pet()

Encephalitis

This doesn't let you annotate a function that only accepts arguments that can fly and swim, which is what the question is asking. It also makes it much harder to call the functions (instead of calling fly() you need to search the feature list for it). – Combustible

Answer 3

If you can't replace the ABC with Protocol, I suggest having both Animal and AnimalProtocol

class AnimalProtocol(Protocol):
    def name(self) -> str: ...

class Animal(ABC, AnimalProtocol):
    @abstractmethod
    def name(self) -> str: ...

class CanFlyProtocol(Protocol):
    def fly(self) -> None: ...

class CanSwimProtocol(Protocol):
    def swim(self) -> None: ...

class Bird(Animal, CanFlyProtocol):
    def fly(self) -> None:
        print("flap wings")

class Penguin(Bird, CanSwimProtocol):
    def name(self) -> str:
        return "penguin"

    def swim(self) -> None:
        print("paddle flippers")


T = TypeVar("T", bound=AnimalProtocol, contravariant=True)

class Petter(Generic[T], ABC):
    @abstractmethod
    def pet(self, a: T) -> None: ...

class CanFlyAndSwimProtocol(AnimalProtocol, CanFlyProtocol, CanSwimProtocol, Protocol):
    pass

class CanFlyAndSwimPetter(Petter[CanFlyAndSwimProtocol]):
    def pet(self, a: CanFlyAndSwimProtocol):
        a.name()
        a.fly()
        a.swim()

CanFlyAndSwimPetter().pet(Penguin())

Problem 1:
We can't completely trust AnimalProtocol to be equivalent to the Animal class. what if other classes have a name() function? pet() will accept them!

To solve this we can add a unique function to the Protocol to make sure it's only implemented by Animal:

class AnimalProtocol(Protocol):
    def name(self) -> str: ...
    def _unique_function_animal_class_only(self) -> None: ...

class Animal(ABC, AnimalProtocol):
    @abstractmethod
    def name(self) -> str: ...

    def _unique_function_animal_class_only(self) -> None:
        """Make sure only this class implements AnimalProtocol"""

This solution can work for you, but consider just leaving it as it was before. The added benefit of absolute type safety comes with the price of clarity.

We should also trust our tests to protect us from this situation.

Problem 2:
When we add more attributes to Animal that are not in AnimalProtocol, we can't use them inside pet() because it only knows about AnimalProtocol but knows nothing about Animal.

We can try to keep Animal and AnimalProtocol synced, but it's a duplication of code and not so practical.

To solve this we can add a assert isinstance(a, Animal) line at the beginning of pet():

class AnimalProtocol(Protocol):
    def name(self) -> str: ...

class Animal(ABC, AnimalProtocol):
    @abstractmethod
    def name(self) -> str: ...

    def say_my_name(self) -> None:
        print(f"My name is {self.name()}")

class CanFlyAndSwimPetter(Petter[CanFlyAndSwimProtocol]):
    def pet(self, a: CanFlyAndSwimProtocol):
        assert isinstance(a, Animal)
        a.name()
        a.fly()
        a.swim()
        a.say_my_name()

This is until we have Intersection type

Kokoruda

Answer 4

Use inheritance and multiple mixins. Python supports multiple inheritance specifically for this purpose.

Kobayashi

Yes, but this is not elegant. Users prefer composition over inheritance. – Senegambia

It is the literal pythonic solution. – Kobayashi

The original code already uses multiple inheritance. What are you suggesting that is different? – Combustible

This approach does not satisfy the OCP. – Hubsher