Circular imports hell

Question

Python is extremely elegant language. Well, except... except imports. I still can't get it work the way it seems natural to me.

I have a class MyObjectA which is in file mypackage/myobjecta.py. This object uses some utility functions which are in mypackage/utils.py. So in my first lines in myobjecta.py I write:

from mypackage.utils import util_func1, util_func2

But some of the utility functions create and return new instances of MyObjectA. So I need to write in utils.py:

from mypackage.myobjecta import MyObjectA

Well, no I can't. This is a circular import and Python will refuse to do that.

There are many question here regarding this issue, but none seems to give satisfactory answer. From what I can read in all the answers:

1. Reorganize your modules, you are doing it wrong! But I do not know how better to organize my modules even in such a simple case as I presented.
2. Try just import ... rather than from ... import ... (personally I hate to write and potentially refactor all the full name qualifiers; I love to see what exactly I am importing into module from the outside world). Would that help? I am not sure, still there are circular imports.
3. Do hacks like import something in the inner scope of a function body just one line before you use something from other module.

I am still hoping there is solution number 4) which would be Pythonic in the sense of being functional and elegant and simple and working. Or is there not?

Note: I am primarily a C++ programmer, the example above is so much easily solved by including corresponding headers that I can't believe it is not possible in Python.

Foreignborn

From what you say, I'd either put MyObjectA in utils.py because some utility functions returns it, or I would put the utility functions that returns a MyObjectA in myobjecta.py. But that's answer 1) – Vertebral

I searched for python circular imports and found several questions. Did you search? – Endorsed

@ Dain: Yes, certainly could do that if the utility functions return only MyObjectA. But what if they also may return MyObjectB or MyObjectC etc. – Foreignborn

@Tom: Yes I did. I summarized my findings in points 1 to 3. – Foreignborn

Can you clarify what's going wrong when you run the code the way you have described it? You say "This is a circular import and Python will refuse to do that", but that's not generally true. Python allows some kinds of circular imports. Packages in this case may be the source of your issue. Are any of the relevant modules being imported from mypackage/__init__.py? If so, removing that import may make the circular imports inside the package work fine. – Coparcenary

Answer 1

There is nothing hackish about importing something in a function body, it's an absolutely valid pattern:

def some_function():
    import logging
    do_some_logging()

Usually ImportErrors are only raised because of the way import() evaluates top level statements of the entire file when called.

In case you do not have a logic circular dependency... , nothing is impossible in python...

There is a way around it if you positively want your imports on top:

From David Beazleys excellent talk Modules and Packages: Live and Let Die! - PyCon 2015, 1:54:00, here is a way to deal with circular imports in python:

try:
    from images.serializers import SimplifiedImageSerializer
except ImportError:
    import sys
    SimplifiedImageSerializer = sys.modules[__package__ + '.SimplifiedImageSerializer']

This tries to import SimplifiedImageSerializer and if ImportError is raised (due to a circular import error or the it not existing) it will pull it from the importcache.

PS: You have to read this entire post in David Beazley's voice.

Amortize

Thank you, I definitely will take the time to see that talk. Nevertheless, importing inside function is not so nice solution to qualify it for 'very pythonic' medal. It hides the module dependency and buries it into code. Why don't do all imports in code then? I miss the logic in it... But yes, it works most of the time. And using try/catch just to circumvent the (IMHO) flawed concept of imports in Python is ... definitely a hack. – Foreignborn

An ImportError is never raised if the module in question is already imported. When importing something, Python checks if it is already imported and simply makes a pointer to it if this is the case. That except statement is only executed when SimplifiedImageSerializer cannot be imported due to it not existing or when a circular import is created. – Satellite

Answer 2

Don't import mypackage.utils to your main module, it already exists in mypackage.myobjecta. Once you import mypackage.myobjecta the code from that module is being executed and you don't need to import anything to your current module, because mypackage.myobjecta is already complete.

Alceste

Is this a good solution? What if I do want to use the utility functions also from the main module. Then I would obscure this dependency by not importing utils into main. Also from the viewpoint of myobjecta.py, using utils.py is just an implementation detail. It introduces potential refactoring chaos when I later decide to change this implementation detail... And how does this break the circular import between myobjecta.py and utils.py? But maybe I misunderstood the point. – Foreignborn

you can import them like this: from mypackage.myobjecta import util_func1, util_func2 since they are already in that package – Alceste

But as I mentioned, using utils from myobjecta is just an implementation detail. What if I later change that. Then I would have to rewrite everything what uses utils and myobjecta at the same time. I personally consider this to be a hack because it forces the user to know the implementation details. This is not a good design. And btw. it does not break the circular import between myobjecta and utils. Or is it? I don't get it. – Foreignborn

Answer 3

What you want isn't possible. There's no way for Python to know in which order it needs to execute the top-level code in order to do what you ask.

Assume you import utils first. Python will begin by evaluating the first statement, from mypackage.myobjecta import MyObjectA, which requires executing the top level of the myobjecta module. Python must then execute from mypackage.utils import util_func1, util_func2, but it can't do that until it resolves the myobjecta import.

Instead of recursing infinitely, Python resolves this situation by allowing the innermost import to complete without finishing. Thus, the utils import completes without executing the rest of the file, and your import statement fails because util_func1 doesn't exist yet.

The reason import myobjecta works is that it allows the symbols to be resolved later, after the body of every module has executed. Personally, I've run into a lot of confusion even with this kind of circular import, and so I don't recommend using them at all.

If you really want to use a circular import anyway, and you want them to be "from" imports, I think the only way it can reliably work is this: Define all symbols used by another module before importing from that module. In this case, your definitions for util_func1 and util_func2 must be before your from mypackage.myobjecta import MyObjectA statement in utils, and the definition of MyObjectA must be before from mypackage.utils import util_func1, util_func2 in myobjecta.

Compiled languages like C# can handle situations like this because the top level is a collection of definitions, not instructions. They don't have to create every class and every function in the order given. They can work things out in whatever order is required to avoid any cycles. (C++ does it by duplicating information in prototypes, which I personally feel is a rather hacky solution, but that's also not how Python works.)

The advantage of a system like Python is that it's highly dynamic. Yes you can define a class or a function differently based on something you only know at runtime. Or modify a class after it's been created. Or try to import dependencies and go without them if they're not available. If you don't feel these things are worth the inconvenience of adhering to a strict dependency tree, that's totally reasonable, and maybe you'd be better served by a compiled language.

Sorenson

Answer 4

Pythonistas frown upon importing from a function. Pythonistas usually frown upon global variables. Yet, I saw both and don't think the projects that used them were any worse than others done by some strict Pythhonistas. The feature does exist, not going into a long argument over its utility.

There's an alternative to the problem of importing from a function: when you import from the top of a file (or the bottom, really), this import will take some time (some small time, but some time), but Python will cache the entire file and if another file needs the same import, Python can retrieve the module quickly without importing. Whereas, if you import from a function, things get complicated: Python will have to process the import line each time you call the function, which might, in a tiny way, slow your program down.

A solution to this is to cache the module independently. Okay, this uses imports inside function bodies AND global variables. Wow!

_MODULEA = None

def util1():
    if _MODULEA is None:
        from mymodule import modulea as _MODULEA

    obj = _MODULEA.ClassYouWant
    return obj

I saw this strategy adopted with a project using a flat API. Whether you like it or not (and I'm not sure about that myself), it works and is fast, because the import line is executed only once (when the function first executes). Still, I would recommend restructuring: problems with circular imports show a problem in structure, usually, and this is always worth fixing. I do agree, though, it would be nice if Python provided more useful errors when this kind of situation happens.

Doy