Trace Python imports

Question

My Python library just changed it's main module name from foo.bar to foobar. For backward compat, foo.bar still exists, but importing it raises a few warnings. Now, it seems some example program still imports from the old module, but not directly.

I'd like to find the erroneous import statement. Is there any tool that allows me to trace imports and find the culprit without wading through all of the code?

Workbook

Answer 1

Start the python interpreter with -v:

$ python -v -m /usr/lib/python2.6/timeit.py
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/lib/python2.6/site.pyc matches /usr/lib/python2.6/site.py
import site # precompiled from /usr/lib/python2.6/site.pyc
# /usr/lib/python2.6/os.pyc matches /usr/lib/python2.6/os.py
import os # precompiled from /usr/lib/python2.6/os.pyc
import errno # builtin
import posix # builtin
# /usr/lib/python2.6/posixpath.pyc matches /usr/lib/python2.6/posixpath.py
import posixpath # precompiled from /usr/lib/python2.6/posixpath.pyc
# /usr/lib/python2.6/stat.pyc matches /usr/lib/python2.6/stat.py
import stat # precompiled from /usr/lib/python2.6/stat.pyc
# /usr/lib/python2.6/genericpath.pyc matches /usr/lib/python2.6/genericpath.py
import genericpath # precompiled from /usr/lib/python2.6/genericpath.pyc
# /usr/lib/python2.6/warnings.pyc matches /usr/lib/python2.6/warnings.py
import warnings # precompiled from /usr/lib/python2.6/warnings.pyc
# /usr/lib/python2.6/linecache.pyc matches /usr/lib/python2.6/linecache.py
import linecache # precompiled from /usr/lib/python2.6/linecache.pyc
# /usr/lib/python2.6/types.pyc matches /usr/lib/python2.6/types.py
import types # precompiled from /usr/lib/python2.6/types.pyc
# /usr/lib/python2.6/UserDict.pyc matches /usr/lib/python2.6/UserDict.py
...

Then just grep for your old module.

Allround

Also, if you don't have access to the Python command line (e.g., the interpreter is embedded), then you can set PYTHONVERBOSE to 1 in the environment to get the same effect. – Scutate

This is a really useful trick. Often times imports work fine in a shell and blow up in a server setup where the paths are different. Using this trick with PYTHONVERBOSE is a lifesaver when you're not in a Python shell. – Viceregal

By the way, higher values of PYTHONVERBOSE give more info. If 1 only tells where things were picked from, a higher number will result in messages showing what paths were tried. Helps a lot when figuring out why a package was not found. – Caseate

This does not provide a stack trace for imports. Which file called import site? Was it timeit.py? Or was it zipimport? In a much more complicated environment I need to know the trace, but cannot find one. – Stancil

Answer 2

edit foo.bar module, add following code:

import pdb
pdb.set_trace()

when foo.bar be imported, program will stop at pdb.set_trace() in pdb mode, where you can debug your code. For example, you can use "w" command to print the full calling stack.

Chaves

Answer 3

I will first introduce the general solution and then show how it can be adapted for this particular use case.

General solution

Nowadays there are easier ways to debug such things utilizing the new functionalities of the python import system. Basically just add your own module finders (MetaPathFinder) to sys.meta_path. Here is an example for a script importing pandas and listing all the succesful imports in imported and unsuccesful imports in could_not_be_imported:

import sys

imported = []
could_not_be_imported = []


class FirstFinder:
    def find_spec(self, modulename, path=None, target=None):
        imported.append(modulename)


class LastFinder:
    def find_spec(self, modulename, path=None, target=None):
        imported.remove(modulename)
        could_not_be_imported.append(modulename)

# setup ("start recording imports")
sys.meta_path.insert(0, FirstFinder())
sys.meta_path.append(LastFinder())

import pandas # import anything here 

# cleanup ("stop recording")
sys.meta_path = [
    x for x in sys.meta_path if not isinstance(x, (FirstFinder, LastFinder))
]

The list containing the succesful imports is then imported:

>>> imported
['pandas',
 'numpy',
 'numpy._globals',
 # ...
 'pandas.io.sql',
 'pandas.io.stata',
 'pandas.io.xml',
 'pandas.util._tester',
 'pandas._version']

and could_not_be_imported:

>>> could_not_be_imported 
['pickle5',
 'org',
 'fcntl',
 'backports_abc',
 'six.moves',
 'backports_abc',
 'backports_abc',
 # ...
 'cloudpickle',
 'numexpr',
 'bottleneck',
 'org',
 'backports_abc',
 'backports_abc',
 'backports_abc']

Note: There can be duplicates in either of the lists and all the modules are listed in order of imports. Modify as fits to your needs.

Specific use case: Finding Erroneous import

Modify the LastFinder to be:

class LastFinder:
    def find_spec(self, modulename, path=None, target=None):
        imported.remove(modulename)
        print(
            "\nMissing module! ",
            f"Tried to import: {modulename}\n",
            "Last few last imports:\n\t",
            "\n\t ".join(imported[-10:]),
        )
        could_not_be_imported.append(modulename)

This will then print, with every missing module, a list of the few last succesfully imported modules and the module you were trying to import. For example:

Missing module!  Tried to import: bottleneck
 Last few last imports:
         pandas.core.ops.common
         pandas.core.ops.docstrings
         pandas.core.ops.mask_ops
         pandas.core.ops.methods
         pandas.core.missing
         pandas.core.array_algos.quantile
         pandas.core.sorting
         pandas.core.arrays.boolean
         pandas.core.arrays.masked
         pandas.core.nanops

In this case this would mean that the pandas.core.nanops was the last succcesful import before the unsuccesful import. Therefore, it would be quite easy to check from pandas.core.nanops.__file__ where the broken import is. (look for "bottleneck" in the file). In my case, I found:

# pandas\core\nanops.py
# ...
bn = import_optional_dependency("bottleneck", errors="warn")

How this works?

The python import system goes through the sys.meta_path to look for a spec finder which will have find_spec method which returns something else than None. If None is returned, next finder in sys.meta_path is used for the import.

Psycho