Circular import dependency in Python [duplicate]

Question

Let's say I have the following directory structure:

a\
    __init__.py
    b\
        __init__.py
        c\
            __init__.py
            c_file.py
        d\
            __init__.py
            d_file.py

In the a package's __init__.py, the c package is imported. But c_file.py imports a.b.d.

The program fails, saying b doesn't exist when c_file.py tries to import a.b.d. (And it really doesn't exist, because we were in the middle of importing it.)

How can this problem be remedied?

Hackneyed

Maybe you could try relative imports? #73352 – Ragouzis

this may help ncoghlan_devs-python-notes.readthedocs.org/en/latest/… – Bergius

also just as a reference, it seems circular imports are allowed on python 3.5 (and probably beyond) but not 3.4 (and probably bellow). – Hedgerow

If you catch the import error, it'll work just fine so long as you don't need to use anything in the other module before the first module finishes importing. – Venule

@CharlieParker This applies specifically to relative imports, according to What's new in 3.5. The relevant issue tracker entry is here. Changes were also made in 3.7 to support some absolute import cases. However, this doesn't prevent AttributeErrors - it enables looking up the partially initialized module in sys.modules, but doesn't resolve time paradoxes. – Protozoan

Answer 1

If a depends on c and c depends on a, aren't they actually the same unit then?

You should really examine why you have split a and c into two packages, because either you have some code you should split off into another package (to make them both depend on that new package, but not each other), or you should merge them into one package.

Jordison

Yes, they could be considered the same package. But if this results in a massively huge file then it's impractical. I agree that frequently, circular dependencies mean the design should be thought through again. But there ARE some design patterns where it's appropriate (and where merging the files together would result in a huge file) so I think it's dogmatic to say that the packages should either be combined or the design should be re-evaluated. – Baleful

Answer 2

You may defer the import, for example in a/__init__.py:

def my_function():
    from a.b.c import Blah
    return Blah()

that is, defer the import until it is really needed. However, I would also have a close look at my package definitions/uses, as a cyclic dependency like the one pointed out might indicate a design problem.

Preface

Sometimes circular references are truly unavoidable. This is the only approach that works for me in these circumstances. – Trudi

Wouldn't this add a lot of overhead in every call of foo ? – Commensurate

@Commensurate - only moderately. Python keeps all imported modules in a global cache (sys.modules), so once a module has been loaded, it won't be loaded again. The code might involve a name look up on each call to my_function, but so does code, which references symbols via qualified names (e.g., import foo; foo.frobnicate()) – Preface

of all the possible solutions here, this is the only one which worked for me. There are absolutely circumstances where a circular reference is "the best" solution - particularly when what you are doing is splitting a set of model objects across multiple files for constraining file sizes. – Huntington

Sometimes circular references are precisely the correct way to model the problem. The notion that circular dependencies are somehow an indication of poor design seems to be more a reflection on Python as a language rather than a legitimate design point. – Browse

Does import in middle of the file like this is considered as bad practice? – Retake

@Retake - I would not consider this a bad practice per se (in particular, since it might be a plain necessity sometimes). I would, however, try to solve the problem by restructuring my module dependencies first, before giving up and using this solution, simply because I myself find it less readable. What's actually considered a bad practice is doing from xxx import * in a local scope. But then, import * is frowned upon even on a module level, and a syntax error anyway in local scopes in "modern" Python, which renders the question mood. – Preface

Answer 3

I've wondered this a couple times (usually while dealing with models that need to know about each other). The simple solution is just to import the whole module, then reference the thing that you need.

So instead of doing

from models import Student

in one, and

from models import Classroom

in the other, just do

import models

in one of them, then call models.Classroom when you need it.

Ekaterinodar

Can you show use what models.py looks like? I don't want to put all the class definitions in one file. I want to create a models.py that imports each class from it's own file. I need to see an example file structure. – Farahfarand

It doesn't need to be one file @ROMS models can be a directory that has an __init__.py file that does the importing from models.classroom. – Ekaterinodar

Note that this only solves the problem if there is no top-level code that runs immediately and tries to access an attribute of models. – Protozoan

Very good point @KarlKnechtel, though it is best practice to avoid doing so. – Ekaterinodar

Answer 4

Circular Dependencies due to Type Hints

With type hints, there are more opportunities for creating circular imports. Fortunately, there is a solution using the special constant: typing.TYPE_CHECKING.

The following example defines a Vertex class and an Edge class. An edge is defined by two vertices and a vertex maintains a list of the adjacent edges to which it belongs.

Without Type Hints, No Error

File: vertex.py

class Vertex:
    def __init__(self, label):
        self.label = label
        self.adjacency_list = []

File: edge.py

class Edge:
    def __init__(self, v1, v2):
        self.v1 = v1
        self.v2 = v2

Type Hints Cause ImportError

ImportError: cannot import name 'Edge' from partially initialized module 'edge' (most likely due to a circular import)

File: vertex.py

from typing import List
from edge import Edge


class Vertex:
    def __init__(self, label: str):
        self.label = label
        self.adjacency_list: List[Edge] = []

File: edge.py

from vertex import Vertex


class Edge:
    def __init__(self, v1: Vertex, v2: Vertex):
        self.v1 = v1
        self.v2 = v2

Solution using TYPE_CHECKING

File: vertex.py

from typing import List, TYPE_CHECKING

if TYPE_CHECKING:
    from edge import Edge


class Vertex:
    def __init__(self, label: str):
        self.label = label
        self.adjacency_list: List[Edge] = []

File: edge.py

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from vertex import Vertex


class Edge:
    def __init__(self, v1: Vertex, v2: Vertex):
        self.v1 = v1
        self.v2 = v2

Notice the absence of the quotes around the type hints. Because of the postponed evaluation of annotations in Python 3.10, these type hints are treated as if they were in quotes.

Quoted vs. Unquoted Type Hints

In versions of Python prior to 3.10, conditionally imported types must be enclosed in quotes, making them “forward references”, which hides them from the interpreter runtime.

In Python 3.7, 3.8, and 3.9, a workaround is to use the following special import.

from __future__ import annotations

Towhee

Answer 5

The problem is that when running from a directory, by default only the packages that are sub directories are visible as candidate imports, so you cannot import a.b.d. You can however import b.d. since b is a sub package of a.

If you really want to import a.b.d in c/__init__.py you can accomplish this by changing the system path to be one directory above a and change the import in a/__init__.py to be import a.b.c.

Your a/__init__.py should look like this:

import sys
import os
# set sytem path to be directory above so that a can be a 
# package namespace
DIRECTORY_SCRIPT = os.path.dirname(os.path.realpath(__file__)) 
sys.path.insert(0,DIRECTORY_SCRIPT+"/..")
import a.b.c

An additional difficulty arises when you want to run modules in c as scripts. Here the packages a and b do not exist. You can hack the __int__.py in the c directory to point the sys.path to the top-level directory and then import __init__ in any modules inside c to be able to use the full path to import a.b.d. I doubt that it is good practice to import __init__.py but it has worked for my use cases.

Taber

Answer 6

I suggest the following pattern. Using it will allow auto-completion and type hinting to work properly.

cyclic_import_a.py

import playground.cyclic_import_b

class A(object):
    def __init__(self):
        pass

    def print_a(self):
        print('a')

if __name__ == '__main__':
    a = A()
    a.print_a()

    b = playground.cyclic_import_b.B(a)
    b.print_b()

cyclic_import_b.py

import playground.cyclic_import_a

class B(object):
    def __init__(self, a):
        self.a: playground.cyclic_import_a.A = a

    def print_b(self):
        print('b1-----------------')
        self.a.print_a()
        print('b2-----------------')

---

You cannot import classes A & B using this syntax

from playgroud.cyclic_import_a import A
from playground.cyclic_import_b import B

---

You cannot declare the type of parameter a in class B __ init __ method, but you can "cast" it this way:

def __init__(self, a):
    self.a: playground.cyclic_import_a.A = a

Luke

Answer 7

Another solution is to use a proxy for the d_file.

For example, let's say that you want to share the blah class with the c_file. The d_file thus contains:

class blah:
    def __init__(self):
        print("blah")

Here is what you enter in c_file.py:

# do not import the d_file ! 
# instead, use a place holder for the proxy of d_file
# it will be set by a's __init__.py after imports are done
d_file = None 

def c_blah(): # a function that calls d_file's blah
    d_file.blah()

And in a's init.py:

from b.c import c_file
from b.d import d_file

class Proxy(object): # module proxy
    pass
d_file_proxy = Proxy()
# now you need to explicitly list the class(es) exposed by d_file
d_file_proxy.blah = d_file.blah 
# finally, share the proxy with c_file
c_file.d_file = d_file_proxy

# c_file is now able to call d_file.blah
c_file.c_blah() 

Recto

modifying global module attributes in a different file like that will quickly lead to a nightmare – Concord