Python type hinting without cyclic imports

Question

I'm trying to split my huge class into two; well, basically into the "main" class and a mixin with additional functions, like so:

main.py file:

import mymixin.py

class Main(object, MyMixin):
    def func1(self, xxx):
        ...

mymixin.py file:

class MyMixin(object):
    def func2(self: Main, xxx):  # <--- note the type hint
        ...

Now, while this works just fine, the type hint in MyMixin.func2 of course can't work. I can't import main.py, because I'd get a cyclic import and without the hint, my editor (PyCharm) can't tell what self is.

I'm using Python 3.4, but I'm willing to move to 3.5 if a solution is available there.

Is there any way I can split my class into two files and keep all the "connections" so that my IDE still offers me auto-completion and all the other goodies that come from it knowing the types?

Platinize

I don't think you should normally need to annotate the type of self, since it's always going to be a subclass of the current class (and any type checking system should be able to figure that out on its own). Is func2 trying to call func1, which isn't defined in MyMixin? Perhaps it should be (as an abstractmethod, maybe)? – Anuria

also note that generally more-specific classes (eg your mixin) should go to the left of base classes in the class definition i.e. class Main(MyMixin, SomeBaseClass) so that methods from the more-specific class can override ones from the base class – Landscapist

I'm not sure how these comments are useful, since they are tangential to the question being asked. velis wasn't asking for a code review. – Sacttler

Answer 1

There isn't a hugely elegant way to handle import cycles in general, I'm afraid. Your choices are to either redesign your code to remove the cyclic dependency, or if it isn't feasible, do something like this:

# some_file.py

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from main import Main

class MyObject(object):
    def func2(self, some_param: 'Main'):
        ...

The TYPE_CHECKING constant is always False at runtime, so the import won't be evaluated, but mypy (and other type-checking tools) will evaluate the contents of that block.

We also need to make the Main type annotation into a string, effectively forward declaring it since the Main symbol isn't available at runtime.

If you are using Python 3.7+, we can at least skip having to provide an explicit string annotation by taking advantage of PEP 563:

# some_file.py

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from main import Main

class MyObject(object):
    # Hooray, cleaner annotations!
    def func2(self, some_param: Main):
        ...

The from __future__ import annotations import will make all type hints be strings and skip evaluating them. This can help make our code here mildly more ergonomic.

All that said, using mixins with mypy will likely require a bit more structure then you currently have. Mypy recommends an approach that's basically what deceze is describing -- to create an ABC that both your Main and MyMixin classes inherit. I wouldn't be surprised if you ended up needing to do something similar in order to make Pycharm's checker happy.

Recurvate

Thanks for this. My current python 3.4 doesn't have typing, but PyCharm was quite happy with if False: as well. – Platinize

The only problem is that it doesn't recognise MyObject as a Django models.Model and thus nags about instance attributes being defined outside of __init__ – Platinize

Here is the corresponding pep for typing. TYPE_CHECKING : python.org/dev/peps/pep-0484/#runtime-or-type-checking – Orten

This works great! You get linting/ type checking of mixin, without circular imports at runtime. Thanks! – Fracas

I still get an issue when doing that and want to specify the return type of a method, for example def func()->Main: pass using Main form the example, if I implement the import like you describe the return type Main is not recognized. It has to be normally imported. – Paternity

What if I want to type hint List[Main] ? – Goldenseal

Thanks - got to this question some 7 years later trying to understand how to do Python type hints for custom classes without creating the equivalent of java's interfaces. – Xe

Answer 2

For people struggling with cyclic imports when importing class only for Type checking: you will likely want to use a Forward Reference (PEP 484 - Type Hints):

When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.

So instead of:

class Tree:
    def __init__(self, left: Tree, right: Tree):
        self.left = left
        self.right = right

you do:

class Tree:
    def __init__(self, left: 'Tree', right: 'Tree'):
        self.left = left
        self.right = right

Ducky

Might be PyCharm. Are you using the newest version? Have you tried File -> Invalidate Caches? – Ducky

Thanks. Sorry, I had deleted my comment. It had mentioned that this works, but PyCharm complains. I resolved using the if False hack suggested by Velis. Invalidating the cache didn't resolve it. It probably is a PyCharm issue. – Sacttler

@JacobLee Instead of if False: you can also from typing import TYPE_CHECKING and if TYPE_CHECKING:. – Jeramyjerba

This does not work if the type resides in another module (at least pycharm doesn't understand it). It would be great if the string could be a fully qualified path. – Frankfurter

This solutions works well in VSCode! Thanks!! – Mazuma

Answer 3

The bigger issue is that your types aren't sane to begin with. MyMixin makes a hardcoded assumption that it will be mixed into Main, whereas it could be mixed into any number of other classes, in which case it would probably break. If your mixin is hardcoded to be mixed into one specific class, you may as well write the methods directly into that class instead of separating them out.

To properly do this with sane typing, MyMixin should be coded against an interface, or abstract class in Python parlance:

import abc


class MixinDependencyInterface(abc.ABC):
    @abc.abstractmethod
    def foo(self):
        pass


class MyMixin:
    def func2(self: MixinDependencyInterface, xxx):
        self.foo()  # ← mixin only depends on the interface


class Main(MixinDependencyInterface, MyMixin):
    def foo(self):
        print('bar')

Mozambique

Well, I'm not saying my solution is great. It's just what I'm attempting to do in order to make the code more manageable. Your suggestion might pass, but this would actually mean just moving the entire Main class to the interface in my specific case. – Platinize

I think that it is the only correct solution. And since the OP wanted Main and MyMixin to be separated in files main.py and mymixin.py respectively, I guess that it necessarily implies creating a third file api.py holding MixinDependencyInterface, doesn’t it? – Hartill

@Platinize typing.Protocol can be used instead of abc.ABC in that you don't actually need to subclass it to register it. It's the proper way to provide interfaces you plan to use, whereas abc.ABC is better for when you provide partially completed implementations i.e. you actually want to subclass it. – Beera

Answer 4

Since Python 3.5, breaking your classes up into separate files is easy.

It's actually possible to use import statements inside of a class ClassName: block in order to import methods into a class. For instance,

class_def.py:

class C:
    from _methods1 import a
    from _methods2 import b

    def x(self):
        return self.a() + " " + self.b()

In my example,

• C.a() will be a method which returns the string hello
• C.b() will be a method which returns hello goodbye
• C.x() will thus return hello hello goodbye.

To implement a and b, do the following:

_methods1.py:

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from class_def import C

def a(self: C):
    return "hello"

Explanation: TYPE_CHECKING is True when the type checker is reading the code. Since the type checker doesn't need to execute the code, circular imports are fine when they occur within the if TYPE_CHECKING: block. The __future__ import enables postponed annotations. This is an optional; without it you must quote the type annotations (i.e. def a(self: "C"):).

We define _methods2.py similarly:

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from class_def import C

def b(self: C):
    return self.a() + " goodbye"

In VS Code, I can see the type detected from self.a() when hovering:

And everything runs as expected:

>>> from class_def import C
>>> c = C()
>>> c.x()
'hello hello goodbye'

Notes on older Python versions

For Python versions ≤3.4, TYPE_CHECKING is not defined, so this solution won't work.

For Python versions ≤3.6, postponed annotations are not defined. As a workaround, omit from __future__ import annotations and quote the type declarations as mentioned above.

Charlet

Answer 5

Rather than forcing oneself to engage in typing.TYPE_CHECKING shenanigans, there is a simple way to avoid circular type-hints: don't use from imports, and use either from __future__ import annotations or string annotations.

# foo.py
from __future__ import annotations
import bar


class Foo:
    bar: bar.Bar

# bar.py
import foo


class Bar:
    foo: "foo.Foo"

This style of import is "lazily evaluated", whereas using from foo import Foo would force Python to run the entire foo module to get the final value of Foo immediately at the import line. It's quite useful if you need to use it at runtime as well e.g. if foo.Foo or bar.Bar needs to be used within a function/method, since your functions/methods should only be called once foo.Foo and bar.Bar can be used.

Beera

What's the best approach if not actually importing the user defined class (because it would cause a circular import error)? For example, if the user defined class is being passed in as an argument to a class in another module. I would have thought the second example but my linter in VS Code flags it as a problem. – Godless

Not sure this works. From what I can remember, there is no such thing as lazy evaluation at import, the whole module is always evaluated even if you only import part of it. – Overcareful

Answer 6

Turns out my original attempt was quite close to the solution as well. This is what I'm currently using:

# main.py
import mymixin.py

class Main(object, MyMixin):
    def func1(self, xxx):
        ...

# mymixin.py
if False:
    from main import Main

class MyMixin(object):
    def func2(self: 'Main', xxx):  # <--- note the type hint
        ...

Note the import within if False statement that never gets imported (but IDE knows about it anyway) and using the Main class as string because it's not known at runtime.

Platinize

I'd expect this to cause a warning about dead code. – Pape

@Phil: yes, at the time I was using Python 3.4. Now there's typing.TYPE_CHECKING – Platinize

Looks stupid, but works with PyCharm. Have my upvote! :) – Wame

Answer 7

I would advice refactoring your code, as some other persons suggested.

I can show you a circular error I recently faced:

BEFORE:

# person.py
from spell import Heal, Lightning

class Person:
    def __init__(self):
        self.life = 100

class Jedi(Person):
    def heal(self, other: Person):
        Heal(self, other)

class Sith(Person):
    def lightning(self, other: Person):
        Lightning(self, other)

# spell.py
from person import Person, Jedi, Sith

class Spell:
    def __init__(self, caster: Person, target: Person):
        self.caster: Person = caster
        self.target: Person = target

class Heal(Spell):
    def __init__(self, caster: Jedi, target: Person):
        super().__init__(caster, target)
        target.life += 10

class Lightning(Spell):
    def __init__(self, caster: Sith, target: Person):
        super().__init__(caster, target)
        target.life -= 10

# main.py
from person import Jedi, Sith

Step by step:

# main starts to import person
from person import Jedi, Sith

# main did not reach end of person but ...
# person starts to import spell
from spell import Heal, Lightning

# Remember: main is still importing person
# spell starts to import person
from person import Person, Jedi, Sith

console:

ImportError: cannot import name 'Person' from partially initialized module
'person' (most likely due to a circular import)

A script/module can be imported only by one and only one script.

AFTER:

# person.py
class Person:
    def __init__(self):
        self.life = 100

# spell.py
from person import Person

class Spell:
    def __init__(self, caster: Person, target: Person):
        self.caster: Person = caster
        self.target: Person = target

# jedi.py
from person import Person
from spell import Spell

class Jedi(Person):
    def heal(self, other: Person):
        Heal(self, other)

class Heal(Spell):
    def __init__(self, caster: Jedi, target: Person):
        super().__init__(caster, target)
        target.life += 10

# sith.py
from person import Person
from spell import Spell

class Sith(Person):
    def lightning(self, other: Person):
        Lightning(self, other)

class Lightning(Spell):
    def __init__(self, caster: Sith, target: Person):
        super().__init__(caster, target)
        target.life -= 10

# main.py
from jedi import Jedi
from sith import Sith

jedi = Jedi()
print(jedi.life)
Sith().lightning(jedi)
print(jedi.life)

order of executed lines:

from jedi import Jedi  # start read of jedi.py
from person import Person  # start AND finish read of person.py
from spell import Spell  # start read of spell.py
from person import Person  # start AND finish read of person.py
# finish read of spell.py

# idem for sith.py

console:

100
90

File composition is key Hope it will help :D

Retrochoir

I'd just like to point out that the question is not about splitting multiple classes into multiple files. It's about splitting a single class into multiple files. Maybe I could refactor this class into multiple classes, but in this case I don't want to. Everything actually belongs there. But it's hard to maintain a >1000 line source, so I split by some arbitrary criteria. – Platinize