Summary
TLDR: How to avoid circular import errors when a base class returns a subclass instance in an importable module?
I have collected some solutions from other locations/questions (see A-D, below) but none is satisfactory IMHO.
Starting point
Based on this and this question, I have the following hypothetical working example as a starting point:
# onefile.py
from abc import ABC, abstractmethod
class Animal(ABC):
def __new__(cls, weight: float):
if cls is Animal:
# Try to return subclass instance instead.
for subcls in [Dog, Cat]:
try:
return subcls(weight)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
@property
@abstractmethod
def weight(self) -> float:
"""weight of the animal in kg."""
...
class Dog(Animal):
def __init__(self, weight: float = 5):
if not (1 < weight < 90):
raise ValueError("No dog has this weight")
self._weight = weight
weight: float = property(lambda self: self._weight)
class Cat(Animal):
def __init__(self, weight: float = 5):
if not (0.5 < weight < 15):
raise ValueError("No cat has this weight")
self._weight = weight
weight: float = property(lambda self: self._weight)
if __name__ == "__main__":
a1 = Dog(34)
try:
a2 = Dog(0.9) # ValueError
except ValueError:
pass
else:
raise RuntimeError("Should have raised Exception!")
a3 = Cat(0.8)
try:
a4 = Cat(25) # ValueError
except ValueError:
pass
else:
raise RuntimeError("Should have raised Exception!")
a5 = Animal(80) # can only be dog; should return dog.
assert type(a5) is Dog
a6 = Animal(0.7) # can only be cat; should return cat.
assert type(a6) is Cat
a7 = Animal(10) # can be both; should return dog.
assert type(a7) is Dog
try:
a8 = Animal(400)
except NotImplementedError:
pass
else:
raise RuntimeError("Should have raised Exception!")
This file runs correctly.
Refactor into importable module, in separate files
I want to have Cat, Dog and Animal as importable classes from the module zoo. To that end, I create a folder zoo, with the files animal.py, dog.py, cat.py, and __init__.py. The file usage.py is kept in parent folder. This is what these files look like:
# zoo/animal.py
from abc import ABC, abstractmethod
from .dog import Dog
from .cat import Cat
class Animal(ABC):
def __new__(cls, *args, **kwargs):
if cls is Animal:
# Try to return subclass instance instead.
for subcls in [Dog, Cat]:
try:
return subcls(*args, **kwargs)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
@property
@abstractmethod
def weight(self) -> float:
"""weight of the animal in kg."""
...
# zoo/dog.py
from .animal import Animal
class Dog(Animal):
def __init__(self, weight: float = 5):
if not (1 < weight < 90):
raise ValueError("No dog has this weight")
self._weight = weight
weight: float = property(lambda self: self._weight)
# zoo/cat.py
from .animal import Animal
class Cat(Animal):
def __init__(self, weight: float = 5):
if not (0.5 < weight < 15):
raise ValueError("No cat has this weight")
self._weight = weight
weight: float = property(lambda self: self._weight)
# zoo/__init__.py
from .dog import Dog
from .cat import Cat
from .animal import Animal
# usage.py
from zoo import Dog, Cat, Animal
a1 = Dog(34)
try:
a2 = Dog(0.9) # ValueError
except ValueError:
pass
else:
raise RuntimeError("Should have raised Exception!")
a3 = Cat(0.8)
try:
a4 = Cat(25) # ValueError
except ValueError:
pass
else:
raise RuntimeError("Should have raised Exception!")
a5 = Animal(80) # can only be dog; should return dog.
assert type(a5) is Dog
a6 = Animal(0.7) # can only be cat; should return cat.
assert type(a6) is Cat
a7 = Animal(10) # can be both; should return dog.
assert type(a7) is Dog
try:
a8 = Animal(400)
except NotImplementedError:
pass
else:
raise RuntimeError("Should have raised Exception!")
This is what is currently not working; the refactoring reintroduces the ImportError (...) (most likely due to a circular import). The problem is that animal.py references dog.py and cat.py, and vice versa.
Possible solutions
Some possibilities are available (some taken from the linked question); here are some options. The code samples only show how relevant parts of the files change.
A: Import modules and move to after Animal class definition
from abc import ABC, abstractmethod
class Animal(ABC):
def __new__(cls, *args, **kwargs):
if cls is Animal:
# Try to return subclass instance instead.
for subcls in [dog.Dog, cat.Cat]: # <-- instead of [Dog, Cat]
try:
return subcls(*args, **kwargs)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
(...)
from . import dog # <-- import module instead of class, and import at end, to avoid circular import error
from . import cat # <-- same
This works.
Disadvantages:
- From
dog.py, theDogclass is really only needed. It's confusing that it's imported completely (though this is considered best practices by some). - Bigger issue: the imports need to be placed at the end of the file, which is definitely bad practice.
B: Move imports inside function
# zoo/animal.py
from abc import ABC, abstractmethod
class Animal(ABC):
def __new__(cls, *args, **kwargs):
from .dog import Dog # <-- imports here instead of at module level
from .cat import Cat # <-- imports here instead of at module level
if cls is Animal:
# Try to return subclass instance instead.
for subcls in [Dog, Cat]:
try:
return subcls(*args, **kwargs)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
(...)
This works too.
Disadvantages:
- Goes against best practice of module-level-only imports.
- If
DogorCatare needed at several locations, the imports need to be repeated.
C: Remove imports and find class by name
# zoo/animal.py
from abc import ABC, abstractmethod
class Animal(ABC):
def __new__(cls, *args, **kwargs):
if cls is Animal:
# Try to return subclass instance instead.
subclasses = {sc.__name__: sc for sc in Animal.__subclasses__()} # <-- create dictionary
for subcls in [subclasses["Dog"], subclasses["Cat"]]: # <-- instead of [Dog, Cat]
try:
return subcls(*args, **kwargs)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
(...)
This also works. In order to avoid creating the dictionary every single time, a registry as shown in this answer could be used as well.
Disadvantages:
- Verbose and barely readable.
- If class name changes, this code breaks.
D: Update dummy variable
# zoo/animal.py
from abc import ABC, abstractmethod
_Dog = _Cat = None # <-- dummies, to be assigned by subclasses.
class Animal(ABC):
def __new__(cls, *args, **kwargs):
if cls is Animal:
# Try to return subclass instance instead.
for subcls in [_Dog, _Cat]: # <-- instead of [Dog, Cat]
try:
return subcls(*args, **kwargs)
except ValueError:
pass
raise NotImplementedError("No appropriate subclass found.")
return super().__new__(cls)
(...)
# zoo/dog.py
from . import animal
from .animal import Animal
class Dog(Animal):
(...)
animal._Dog = Dog # <-- update protected variable
# zoo/cat.py analogously
This works as well.
Disadvantages:
- Unclear to reader, what the
_Dogand_Catvariables inzoo/animal.pyrepresent. - Coupling between files; change/use of module's "protected" variables from outside.
E: a better solution??
None of A-D is satisfying, in my opinion, and I'm wondering if there's another way. This is where you come in. ;) There might not be another way - in that case I'm curious to hear what your preferred approach would be, and why.
Many thanks
Animalclass returns an instance of one of its known subclasses. So only if the module containingDoghas been loaded, then theDogclass will be available. Is it enough for your requirements or do you want all the subclasses to be available fromAnimal? – EpistyleDogavailable to from withinanimal.py, otherwise it will not know what the name 'Dog' means. (To answer your question - in this case only a subset of all possible subclasses must be available toAnimal, and they must be checked in a specific order.) – Lewak