Why python finds module instead of package if they have the same name?

Question

Here is my directory structure:

/home/dmugtasimov/tmp/name-res
    root
        tests
            __init__.py
            test_1.py
        __init__.py
        classes.py
        extra.py
        root.py

File contents: root/tests/_init_.py

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                             '../..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)
# These imports are required for unittest to find test modules in package properly
from root.tests import test_1

root/tests/test_1.py

import unittest
from root.classes import Class1
class Tests(unittest.TestCase):
    pass

root/_init_.py - empty
root/classes.py

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)

print 'sys.path:', sys.path
print 'BEFORE: import root.extra'
import root.extra
print 'AFTER: import root.extra'

class Class1(object):
    pass

class Class2(object):
    pass

root/extra.py

class Class3(object):
    pass

root/root.py

import os
import sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)
from classes import Class2

---

I get following output:

$ python -m unittest tests.test_1
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
 File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
 File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
 File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
 File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
 File "/usr/lib/python2.7/unittest/loader.py", line 128, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
 File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
 File "tests/__init__.py", line 9, in <module>
    from root.tests import test_1
 File "/home/dmugtasimov/tmp/name-res/root/tests/__init__.py", line 9, in <module>
    from root.tests import test_1
 File "/home/dmugtasimov/tmp/name-res/root/tests/test_1.py", line 3, in <module>
    from root.classes import Class1
 File "/home/dmugtasimov/tmp/name-res/root/classes.py", line 9, in <module>
    import root.extra
 File "/home/dmugtasimov/tmp/name-res/root/root.py", line 6, in <module>
    from classes import Class2
ImportError: cannot import name Class2

It turns out the problem is the order used by python interpreter to search for package or module:

$ python -vv -m unittest tests.test_1
…skipped...
import root.classes # precompiled from /home/dmugtasimov/tmp/name-res/root/classes.pyc
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
# trying /home/dmugtasimov/tmp/name-res/root/root.so
# trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so
# trying /home/dmugtasimov/tmp/name-res/root/root.py
# /home/dmugtasimov/tmp/name-res/root/root.pyc matches /home/dmugtasimov/tmp/name-res/root/root.py
…skipped...

---

According to python documentation http://docs.python.org/2/tutorial/modules.html#the-module-search-path: “When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path.”

This means that python should look at sys.path index 0 entry, get path '/home/dmugtasimov/tmp/name-res' and find package named root and then search module named extra in this package. But instead it searches in /home/dmugtasimov/tmp/name-res/root/ directory for module root and then tries to find something named extra in it. What does it happen? Does not it contradict official documentation? Or are rules for searching packages different than for modules? If so, are these rules covered somewhere in documentation?

UPDATE

I put it here for better formatting.
For futher investigation do the following:

1. Remove root.pyc
2. Rename root.py to root2.py
3. Run python -vv -m unittest tests.test_1

# trying /home/dmugtasimov/tmp/name-res/root/root.so
# trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so
# trying /home/dmugtasimov/tmp/name-res/root/root.py
# trying /home/dmugtasimov/tmp/name-res/root/root.pyc
# trying /home/dmugtasimov/tmp/name-res/root/extra.so
# trying /home/dmugtasimov/tmp/name-res/root/extramodule.so
# trying /home/dmugtasimov/tmp/name-res/root/extra.py

It appears python disregard sys.path only for initial 4 tries.

UPDATE 2

Simplified version:

/home/dmugtasimov/tmp/name-res3/xyz
    __init__.py
    a.py
    b.py
    t.py
    xyz.py

Files init.py, b.py and xyz.py are empty
File a.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in a.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in a.py'

print 'sys.path:', sys.path
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'

File t.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in t.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in t.py'

import xyz.a

Run:

python a.py

Output:

sys.path is modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
AFTER import xyz.b

Run:

python -vv a.py

Output:

import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
# /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.py
# /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py
import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc

Run:

python t.py

Output:

sys.path is modified in t.py
sys.path is NOT modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
Traceback (most recent call last):
  File "t.py", line 9, in <module>
    import xyz.a
  File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module>
    import xyz.b
ImportError: No module named b

Run:

python -vv t.py

Output:

import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
# /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/a.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/amodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/a.py
# /home/dmugtasimov/tmp/name-res3/xyz/a.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/a.py
import xyz.a # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/a.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/os.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/osmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/os.py
# trying /home/dmugtasimov/tmp/name-res3/xyz/os.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/sys.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/sysmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/sys.py
# trying /home/dmugtasimov/tmp/name-res3/xyz/sys.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
# /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc
#   clear[2] __file__
#   clear[2] __package__
#   clear[2] sys
#   clear[2] ROOT_DIRECTORY
#   clear[2] __name__
#   clear[2] os
sys.path is modified in t.py
sys.path is NOT modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
Traceback (most recent call last):
  File "t.py", line 9, in <module>
    import xyz.a
  File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module>
    import xyz.b
ImportError: No module named b

As you see sys.path is the same for both cases:

sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

But the behaviour is different. For a.py python searches for package xyz first, and them for module b in it:

import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
# /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.py
# /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py
import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc

In other words:

1. Search PACKAGE xyz in directory sys.path[0] -> FOUND
2. Search module b in PACKAGE xyz -> FOUND
3. Continue execution

For t.py it searches for moduel xyz in the same directory as a.py itself and then fails to find module b in module xyz:

# trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
# /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc

In other words:

1. Search MODULE xyz in directory in the same directory as a.py (or sys.path[1] ?) -> FOUND
2. Search MODULE b in MODULE xyz -> NOT FOUND
3. ImportError

So it looks like if "import xyz.b" bahaves different depending on how a.py was initially loaded as a script or imported from another module.

UPDATE 3

I filed proposal for documentation fix: http://bugs.python.org/issue16891

UPDATE 4

The reason for behavior described in UPDATE 2 is completely clear for me now.

http://docs.python.org/2/tutorial/modules.html#intra-package-references

6.4.2. Intra-package References

The submodules often need to refer to each other. For example, the surround module might use the echo module. In fact, such references are so common that the import statement first looks in the containing package before looking in the standard module search path. Thus, the surround module can simply use import echo or from echo import echofilter. If the imported module is not found in the current package (the package of which the current module is a submodule), the import statement looks for a top-level module with the given name.

For "python a.py" "a" is not considered as module in package, but for "python t.py" "a" is considered as a module in package "xyz". Therefore in first case it searches according sys.path, but in the second case it searches inside the same package (namely "xyz") for module named "xyz" (in other words "xyz.xyz")

You can easily see if change a.py like this:

File a.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in a.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in a.py'

print 'sys.path:', sys.path
print '__package__', __package__
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'

My output was:

~/tmp/name-res3/xyz $ python a.py
sys.path is modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
__package__ None
BEFORE import xyz.b
AFTER import xyz.b
~/tmp/name-res3/xyz $ python t.py
sys.path is modified in t.py
sys.path is NOT modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
__package__ xyz
BEFORE import xyz.b
Traceback (most recent call last):
  File "t.py", line 9, in <module>
    import xyz.a
  File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 12, in <module>
    import xyz.b
ImportError: No module named b

Thanks to @J.F. Sebastian for pointing out the right place of documentation.

UPDATE 5

It seems that there is another issue. If interested, please, follow comments here: http://bugs.python.org/issue16891

Jackquelin

For futher investigation do the following: 1. Remove root.pyc 2. Rename root.py to root2.py 3. Run python -vv -m unittest tests.test_1 # trying /home/dmugtasimov/tmp/name-res/root/root.so # trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so # trying /home/dmugtasimov/tmp/name-res/root/root.py # trying /home/dmugtasimov/tmp/name-res/root/root.pyc # trying /home/dmugtasimov/tmp/name-res/root/extra.so # trying /home/dmugtasimov/tmp/name-res/root/extramodule.so # trying /home/dmugtasimov/tmp/name-res/root/extra.py It appears python disregard sys.path only for initial 4 tries. – Jackquelin

Please fix (edit) your list of directory structure. The file "root.py" is on the same level as "root" directory but you write "root/root.py". The same is about more files. – Inexcusable

You really shouldn’t modify sys.path, especially not in library code. Your root is a package, so there is no problem with using absolute or relative imports when the __main__ script has the root package in its importable path. – Lucan

Let's assume that I accept that I should NOT modify sys.path. But I did modify it (I have a reason for that) and I expect python to behave as documented (if it is documented). The question is "Is it documented and where?" The question is not "Should I modify sys.path?" ;) – Jackquelin

BTW, sys.path modification in root/classes.py may be removed, it is not the root issue. – Jackquelin

Yes, sys.path modifications can be removed because PYTHONPATH is sufficient. It can't control what is the first on sys.path, that can be the current dir (empty string) first and maybe some "site-packages/distribute-.egg". If your current dir is not side your project "xyz" it makes no problem. However it looks Python 2 tries first to import *relative to the current file and than from sys.path. It is not important for import commands in any file if the file is being imported or executed. The directory of the file is important and that the dir of main file is automatically on sys.path. – Inexcusable

Answer 1

I simplified the example from the question to demostrate that only four solutions are possible:

• Explicit relative import from . import some_module or with more commas from ..
• Relative import (without "xyz." if used in the package "xyz")
• Absolute import using from __future__ import absolute_import (or use Python 3)
• Never repeat the top level importable name of the package in any path inside it.

What solution is the best? It depends on personal preference of Python 2 or 3. Only the last one is nice and universal for all Pythons. It was really a useful question.

---

xyz/tests/__init__.py: import xyz.tests.t

xyz/tests/t.py:

import sys
print('sys.path = %s' % sys.path) # see that the parent of "xyz" is on sys.path
print("importing xyz.tests")
import xyz.a

xyz/a.py:

# solution A: absolute_import by __future__  (or use Python 3)
#from __future__ import absolute_import
print("importing xyz.a")
# solution B: explicit relative import
#from . import b    # and remove "import xyz.b"
# solution C: relative import (not recommended)
#import b           # and remove "import xyz.b"
import xyz.b

xyz/b.py: print("imported xyz.b")

xyz/xyz.py: print("Imported xyz.xyz !!!")

xyz/__init__.py: empty file

---

Everything possible fails, e.g.

parent_of_xyz=...  # The parent directory of "xyz" - absolute path
cd $parent_of_xyz
python -m xyz.tests.t
PYTHONPATH=$parent_of_xyz/xyz python -m unittest tests
PYTHONPATH=$parent_of_xyz     python xyz/tests/t.py

with messages like

Imported xyz.xyz  !!!
...
ImportError...

If any solution is applied (uncommented), all three examples work.

It can be more simplified without using any subdirectory.

EDIT: I tried yesterday many tests but I wrote it inconsitently from different versions. Excuse me that it was unreproducible from the answer. Now it is fixed.

Inexcusable

1. Explicit relative import fails in __main__ (a module ran as a script); 2. relative import shouldn't be used at all; 3. from __future__ import absolute_import is not necessary; 4. you can't avoid repeating some partial names (you don't know what other packages are available on sys.path) – Taenia

@J.F.Sebastian: Only the same name of the package being writed should be avoided in that package. It is an easy restriction althought it is not a well known restriction. The absolute import without _future_ doesn't work correctly in the case above even in Python 2.7, where the feature should be mandatory. Yes, simple relative imports are a bad practice and it is mentioned only for completness that no other good or bad solution doesn't exist. – Inexcusable

Is it a typo here (the last char)? import xyz.tests.t` – Jackquelin

@hynekcer, it looks like your simplification does not correctly reflect initial problem, because I get "ImportError: No module named tests" once I run "python -m unittest tests.t". Another point is that PYTHONPATH may not be inserted at index 0 of sys.path, so it changes search sequence. – Jackquelin

I fixed my bugs caysed by trying many test yesterday. It looked OK for me but I copied my answer inconsitently from different versions during testing. I'm sorry. – Inexcusable

@J.F.Sebastian: Well, 4th solution for me seems like workaround for builtin artificial restriction of python (poor language design, actually, because there is no explicit way to distinct module and package in import statement or a specified order of package vs module name resolution). – Jackquelin

Answer 2

Do not modify sys.path it leads to the issues when the same module is available under different names. See Traps for the Unwary.

Use absolute or explicit relative imports instead in the code and run your scripts from the project directory. Run the tests using the full name:

$ python -munittest root.tests.test_1

Some packages do modify sys.path internally e.g., see how twisted uses _preamble.py or pypy's autopath.py. You could decide whether their downsides (introduction of obscure import problems) are worth the convience (more ways to run your scripts are allowed). Avoid modifying sys.path in a code that is used as a library i.e., limit it to test modules and your command-line scripts.

Taenia

You are right that marginal things in the question can be improved, but the core of the is correct and remains unanswered also by me. It seems really not good docummented in Python 2. Please see my answer. – Inexcusable

@hynekcer: My answer is about the core of the issue: "how to fix importing problems" and the answer: "Use absolute or explicit relative imports instead in the code and run your scripts from the project directory." – Taenia

The absolute import without _future_ directive (or without Python 3) and running from the project directory (or from whatever possible directory) doesn't work, as it is demonstrated in the simplified example. The main project directory from where is the package imported can be completely outside. – Inexcusable

@hynekcer: Thank you for pointing this out: "but the core of the is correct and remains unanswered" – Jackquelin

@hynekcer: yes, in your example absolute import doesn't work. From the docs (Python 2.x): "the import statement first looks in the containing package before looking in the standard module search path" i.e., import xyz.b in xyz/a.py will try to import xyz/xyz.py first instead of xyz/b.py. Renaming xyz.py (as you suggested) solves the issue. – Taenia

Yes, you mean an absolute import without previous from __future__ import absolute_import, which disables any implicit relative searching. – Inexcusable

@J.F.Sebastian: It seems that you found behavior specification buried in the documentation: "From the docs (Python 2.x): "the import statement first looks in the containing package before looking in the standard module search path". Of course, it should not be done as a BTW-note like it is done now, because does not really catch attention of a reader. Thank you very much. – Jackquelin