Tool for pinpointing circular imports in Python/Django?

Question

I have a Django app and somewhere in it is a recursive import that is causing problems. Because of the size of the app I'm having a problem pinpointing the cause of the circular import.

I know that the answer is "just don't write circular imports" but the problem is I'm having a hard time figuring out where the circular import is coming from, so ideally a tool that traced the import back to its origin would be ideal.

Does such a tool exist? Barring that, I feel like I am doing everything I can to avoid circular import problems -- moving imports to the bottom of the page if possible, moving them inside of functions rather than having them at the top, etc. but still running into problems. I'm wondering if there are any tips or tricks for avoiding them altogether.

To elaborate a bit...

In Django specifically when it encounters a circular import, sometimes it throws an error but sometimes it passes through silently but results in a situation where certain models or fields just aren't there. Frustratingly, this often happens in one context (say, the WSGI server) and not in another (the shell). So testing in the shell something like this will work:

Foo.objects.filter(bar__name='Test')

but in the web throws the error:

FieldError: Cannot resolve keyword 'bar__name' into field. Choices are: ...

With several fields conspicuously missing.

So it can't be a straightforward problem with the code since it does work in the shell but not via the website.

Some tool that figured out just what was going on would be great. ImportError is maybe the least helpful exception message ever.

Burmese

python -vv will help finding recursive imports. Example: pastebin.com/3HpYgeC2 – Thrasonical

Is there any way to organize this output in a way so I can see what was called? Seems like this would only work for hard circular import problems not the soft ones like I'm having... – Burmese

(I'm not sure this will help you at all because i'm not sure how to use it in your WSGI environment). Anyway, it can help with "soft"/"runtime" import problems. Like line 627 of the paste where I call manually "import django": it shows all files it tries. I just tested "import django.db" and it shows all files it tries for all modules included by django.db. I don't know but would like to know an existing way to improve the output because it's a pain i totally agree !! – Thrasonical

The silent failure is because you have multiple modules with the same name. Then, python import order (based on pythonpath) is the reference. Oh, when/if you change the name, make sure you remove the .pyc too :) (it happened to me several times) – Knownothing

Answer 1

The cause of the import error is easily found, in the backtrace of the ImportError exception.

When you look in the backtrace, you'll see that the module has been imported before. One of it's imports imported something else, executed main code, and now imports that first module. Since the first module was not fully initialized (it was still stuck at it's import code), you now get errors of symbols not found. Which makes sense, because the main code of the module didn't reach that point yet.

Common causes in Django are:

1. Importing a subpackage from a totally different module,

   e.g. from mymodule.admin.utils import ...

   This will load admin/__init__.py first, which likely imports a while load of other packages (e.g. models, admin views). The admin view gets initialized with admin.site.register(..) so the constructor could start importing more stuff. At some point that might hit your module issuing the first statement.

   I had such statement in my middleware, you can guess where that ended me up with. ;)

2. Mixing form fields, widgets and models.

   Because the model can provide a "formfield", you start importing forms. It has a widget. That widget has some constants from .. er... the model. And now you have a loop. Better import that form field class inside the def formfield() body instead of the global module scope.

3. A managers.py that refers to constants of models.py

   After all, the model needs the manager first. The manager can't start importing models.py because it was still initializing. See below, because this is the most simple situation.

4. Using ugettext() instead of ugettext_lazy.

   When you use ugettext(), the translation system needs to initialize. It runs a scan over all packages in INSTALLED_APPS, looking for a locale.XY.formats package. When your app was just initializing itself, it now gets imported again by the global module scan.

   Similar things happen with a scan for plugins, search_indexes by haystack, and other similar mechanisms.

5. Putting way too much in __init__.py.

   This is a combination of points 1 and 4, it stresses the import system because an import of a subpackage will first initialize all parent packages. In effect, a lot of code is running for a simple import and that increases the changes of having to import something from the wrong place.

The solution isn't that hard either. Once you have an idea of what is causing the loop, you remove that import statement out of the global imports (on top of the file) and place it inside a function that uses the symbol. For example:

# models.py:
from django.db import models
from mycms.managers import PageManager

class Page(models.Model)
    PUBLISHED = 1

    objects = PageManager()

    # ....


# managers.py:
from django.db import models

class PageManager(models.Manager):
    def published(self):
        from mycms.models import Page   # Import here to prevent circular imports
        return self.filter(status=Page.PUBLISHED)

In this case, you can see models.py really needs to import managers.py; without it, it can't do the static initialisation of PageManager. The other way around is not so critical. The Page model could easily be imported inside a function instead of globally.

The same applies to any other situation of import errors. The loop may include a few more packages however.

Sarsenet

this answer would have been better with a clear example of the traceback and detecting a real circular import to see how that works. I'm staring at a traceback and I know it's a circular import but I'm still digging through 5+ files to figure out exactly where the import is incorrect. – Dissepiment

Everyone is writing 'check your code for circular import', and finally this answer turns out to be savior pointing the actual loading behavior of 'ugettext'. Its a Big THANK YOU. – Lefthand

ugettext got me too. Big props! – Egypt

What about not importing the Page model in the manager? Manager itself has its attribute model . – Isoelectronic

What I can't understand, is how to follow an ImportError trace. Normally, in an exception, I can logically follow the trace, stepping back through the code. I can see where each catch was & identify the line of code where it was raised to follow it back to find the problem, but the trace in an ImportError seems completely disjoint to me. From 1 item to the next, there seems to be no connection. I can see different lines that import the same package, but I cannot follow how I get from 1 to the other, & in some cases, it seems like the same flow, but no exception. How do I follow the trace? – Retrograde

Answer 2

One of the common causes of circular imports in Django is using foreign keys in modules that reference each other. Django provides a way to circumvent this by explicitly specifying a model as a string with the full application label:

# from myapp import MyAppModel  ## removed circular import

class MyModel(models.Model):
    myfk = models.ForeignKey(
        'myapp.MyAppModel',  ## avoided circular import
        null=True)

See: https://docs.djangoproject.com/en/dev/ref/models/fields/#foreignkey

Seeing

If I could boost this by 10 points I would. This is a HUGE life saver. – Paillette

This may be obvious but, but note when you use this method, it means you should not do the import at the top of the models.py file. So, start by looking at the line causing the ImportError, if it is a from myapp import MyAppModel, remove that line and instead do as above, put 'myapp.MyAppModel' in the foreign key. – Caliginous

I've edited the code, displaying a commented line to provide clarity based on @Rob's comment. – Seeing

Answer 3

What I normally do when I encounter an import error is to work my way backwards. I'd get some "cannot import xyz from myproject.views" error, even though xyz exists just fine. Then I do two things:

• I grep my own code for every import of myproject.views and make a (mental) list of modules that import it.

• I check if I import one of those matching modules in views.py in turn. That often gives you the culprit.

A common spot where it can go wrong is your models.py. Often central to what you're doing. But make sure you try to keep your imports pointing AT models.py instead of away from it. So import models from views.py, but not the other way around.

And in urls.py, I normally import my views (because I get a nice immediate import error when I make a mistake that way). But to prevent circular import errors, you can also refer to your views with a dotted path string. But this depends on what you're doing in your urls.py.

A comment regarding placement of imports: keep them at the top of the file. If they're spread out you'll never get a clear picture of which module imports what. Just putting them all at the top (nicely sorted) could already help you pinpoint problems. Only import inside functions if necessary for solving a specific circular import.

And make your imports absolute instead of relative. I mean "from myproject.views import xyz" instead of "from views import xyz". Making it absolute combined with sorting the list of imports makes your imports more clear and neat.

Felicity

Answer 4

Just transforming the comment above in an answer...

If you have circular import, python -vv does the trick. Other way would be to overload the module loader (there's a link somewhere but I can't find it just now). Update: you can probably do it with the ModuleFinder

The silent failure happens because you have multiple modules with the same name. Then, python import order (based on pythonpath) is the reference. Oh, when/if you change the name, make sure you remove the .pyc too :) (it happened to me several times)

Knownothing

Answer 5

It can help to visualize the module dependencies using pyreverse.

Install pylint (pyreverse is integrated in pylint) and graphviz (for generating png images):

apt-get install graphviz
pip install pylint

Then generate image with modules dependency graph for modules in <folder>:

pyreverse -o png <folder>

Ideally the dependency graph should flow from bottom up without any circles:

Kugler

I tried this and if failed due to... circular import. Not super helpful for solving a circular import problem – Foxhole

File "/usr/local/lib/python3.7/site-packages/pylint/pyreverse/diagrams.py", line 178, in extract_relationships if value is astroid.YES: AttributeError: module 'astroid' has no attribute 'YES' – Skiplane