Summary
Problems such as this occur when a Python source file in the project has the same name as some external library module (either in the standard library or an installed third-party package). When attempting to import
from the external library (which requires using an absolute import), the project's own module is found instead, causing various errors depending on the exact details.
The easiest way to fix the problem, in ordinary cases, is to rename the affected file. It may also be necessary to locate and delete any corresponding .pyc
files. (This is perfectly safe; the files are just a cache of the work that was previously done to translate Python source code into bytecode for the Python virtual machine, analogous to .class
files in Java.)
When the project's own module is preferred, this is because of how Python searches for module source code for absolute imports. Depending on exactly how Python is started, the module search path defined in sys.path
will usually start with a path that is within the current project. Python will look there first, before looking in any standard library or third-party library folders. This search process does not care about the folder where the importing module is located; any relative paths are relative to the process' current working directory, not the importing module. (However, the standard library paths will normally be absolute paths, and near the end of sys.path
, anyway.)
AttributeError
An AttributeError
occurs because the project's own module simply doesn't define the function, class etc. that the calling code wants to use from the external library module. As a result, the 'module' object
that represents the project's module has no attribute
with the specified name, so that's exactly what the error message claims.
In rare, unfortunate cases, the project's own module might happen to define something with the same name such that it does something different. This can cause any kind of exception or other logical error. For example, if the first example from the question is modified:
import requests
def get():
pass
res = requests.get('http://www.google.ca')
print(res)
now a TypeError
will be raised instead, because the code will attempt to call the locally-defined get
function with the wrong number of arguments.
ImportError
Using a specific from-import causes the error to be reported differently because the problem is now detected during the importing process itself. In the example in the question, from requests import get
means, instead of creating a new global name requests
which names the module, there should be a new global name get
which names the function from that module (the one which would be referred to as requests.get
after a plain import). This requires looking up the get
attribute from that module; but since the wrong module was loaded, that attribute lookup fails. More recent versions of Python report this as a probable circular import.
Other import attempts can cause different ImportError
s which complain that the imported module "is not a package". This is self-explanatory: in the example in the question, requests
is an ordinary module (because it is defined by a source code file, requests.py
), but the desired requests
- defined by the third-party library - is a package (defined by several files inside a requests
folder elsewhere, including an __init__.py
which defines some top-level package contents that are not modules, like the get
function).
NameError
Using a star-import, like from requests import *
, will not fail directly - it creates global names for all the module contents. However, since the module in the question example doesn't contain anything named get
, its attempt to star-import itself will not define that name either. Thus, a NameError
occurs - the normal error for trying to use a global name that doesn't exist.
Aggravating factors
Python's default import
system is based on names, not file paths, and Python does not distinguish between "(driver) script files" and "library (module) files". Any Python source code can be import
ed. Although Python caches module imports, the main script will not generally be in this cache, which means it is perfectly capable of attempting to import
itself. This is what happens in the example in the question: since requests
was not already import
ed, there is nothing in the cache, so the attempt in the driver script (named requests.py
) to import requests
will search for something to import, and find the driver script (i.e., its own source file).
This only causes a problem at import time, if the module tries to use that import in its own initialization, e.g. by doing a from-import. Otherwise, the problem will be deferred, resulting in (typically) AttributeError
or NameError
when trying to use the functionality. See also:
Whenever a module is loaded (i.e., imported from its source, rather than using the cache), its own top-level import
statements run, causing more imports, recursively. Any of these indirect imports could potentially find the local code. The Python standard library is not a package, and mostly uses absolute import to refer to its other components. For example, as pointed out in Dave Rove's answer, attempting to import the standard library decimal
module could fail when tried from a source file named numbers.py
, or within a project that has such a source file.
In one especially pernicious case, having a file named token.py
in a project (or the current working directory, when starting up Python in interactive mode) causes the interactive help to break:
$ touch token.py
$ python
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/_sitebuiltins.py", line 102, in __call__
import pydoc
File "/usr/lib/python3.8/pydoc.py", line 66, in <module>
import inspect
File "/usr/lib/python3.8/inspect.py", line 40, in <module>
import linecache
File "/usr/lib/python3.8/linecache.py", line 11, in <module>
import tokenize
File "/usr/lib/python3.8/tokenize.py", line 35, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/current/working directory/token.py)
The traceback tells us all we need to know: calling help
triggers a deferred import of the standard library pydoc
, which indirectly attempts to import the standard library token
, but finds our token.py
which doesn't contain the appropriate name. In older versions of Python, it was even worse: tokenize
would do a star-import from token
, and then its top-level code would try to use a name defined there, resulting in NameError
- and a stack trace not mentioning the file name token.py
.
Python can import from cached bytecode files (.pyc
) even if the corresponding source file is renamed or deleted. This is intended to speed up imports, but it creates an extra step to keep code directories "clean". After renaming any .py
file to resolve a problem like this, make sure to check the bytecode cache as well.
Solving the problem
Of course, the standard advice is simple and straightforward: just rename the .py
files that were mistakenly imported, and delete any cached bytecode .pyc
files for those modules. By default, in Python 3.x, the .pyc
files for any .py
files within a folder will be put into a subfolder with the special name __pycache__
; in 2.x, they simply appeared alongside the corresponding .py
files.
However, while this approach quickly solves the immediate problem for most people encountering it, the advice does not scale very well. There are a lot of potentially problematic names; while most projects will not be interested in including a source file named e.g. os.py
, some other names might be more desirable, or harder to work around. So, here are some other useful techniques:
Controlling sys.path
Of course, since the problem is caused by sys.path
specifying that absolute imports should look in the current project first, it can be avoided by just changing the sys.path
.
The problematic path is described in the documentation as a "potentially unsafe path". When using the interactive Python prompt, this will be an empty string (a relative path equivalent to '.'
) - i.e., the current working directory for the Python process, reflecting any changes made by using e.g. os.chdir
. For driver scripts started normally (python driver.py
), it will be the directory where the script is located, as an absolute path (not necessarily the current working directory, since a path could be specified on the command line, like python path/to/driver.py
). For modules run using the -m
command-line flag, it will be the initial current working directory (not necessarily where the module is located, but also an absolute path that will not be affected by os.chdir
).
To avoid having this path in sys.path
, do one of the following:
In Python 3.11 and up, set the PYTHONSAFEPATH
environment variable, or use the -P
command-line option to Python.
In Python 3.4 and up, use the -I
command-line option to start in isolated mode. This, however, has several other effects: it will ignore environment variables like PYTHONPATH
and PYTHONHOME
, and it will also skip adding the user-specific site-packages directory to the path (therefore the code will not have access to third-party libraries that were installed using the --user
option for Pip).
If all else fails, consider manually manipulating sys.path
. This is messy and error-prone, but sys.path
is just an ordinary list of strings with file paths, and modifying its contents will affect future import
s. (Do not try to replace the list object; this will not work, because Python does not look for it by the name sys.path
, but uses a hard-coded internal reference. It is not possible to destroy or replace that object from Python; sys.path
is just a name that is initialized to refer to it.)
Keep in mind that removing the "unsafe" path will prevent intentional absolute imports from working within the package. This is inconvenient for some small projects, but also a good reason to learn proper package organization. Speaking of which:
Using relative imports within packages
A well-organized Python project will typically consist of one or more (usually just one) packages, stored in subfolders of the main project folder, plus one or more driver scripts placed outside of the package subfolders. The driver scripts will typically use a single absolute import to access the package functionality, while implementing some simple wrapper logic (e.g. to parse command-line arguments, format and report otherwise-uncaught exceptions, etc.). The package, meanwhile, will use relative imports throughout for its own content, and absolute import only where necessary to access other packages (the standard library and third-party dependencies).
For example, a project might be organized like:
project
├── src
│ └── my_package
│ └── x.py
│ └── y.py
│ └── z.py
└── driver.py
Code in driver.py
will use absolute import into the package, like:
from my_package.x import entry_point
if __name__ == '__main__':
entry_point()
(If logic is needed to parse the command-line arguments, it should normally go in the driver rather than in the package's code.)
Then, code inside the package will use relative imports - so x.py
might contain something like:
from .y import first_thing
from .z import second_thing
def entry_point():
first_thing()
second_thing()
This gets the best of both worlds: the initial absolute import sets up the top-level package so that relative imports will work, and the relative imports will avoid depending on the sys.path
configuration. Even without taking steps to configure sys.path
, it will typically include the folder with the driver scripts, but not any package folders; thus, this also automatically avoids import path conflicts. (An absolute import won't find the package contents unless it specifies the corresponding package path; but typically when it does, importing from the current package was intentional.)
This also avoids setting traps for the next project which has this one as a dependency. Say for example that we implement and publish an API
package, and someone else writes Client
which has API
as a dependency. Since API
code will use relative import for other API
functionality (say, from . import functionality
), the Client
project's own functionality.py
won't cause a problem.
For the initial absolute import to work, of course, the top-level package folder needs to be mentioned in sys.path
. However, this is normally accomplished by installing the package, so it does not cause a problem.
Disabling bytecode caching
If a file needs to be renamed, avoiding problems with .pyc
files will be easier if they simply don't exist in the first place. They are not necessary, after all; they are, again, simply intended to speed up imports on subsequent runs of the program.
To suppress .pyc
file generation, use the -B
command-line option or set the PYTHONDONTWRITEBYTECODE
environment variable. (There is no built-in solution to delete all existing .pyc
files, but this is easy enough to implement by hand using e.g. the shutil
standard library module.)
Linting to avoid problematic names
Consider using third-party tools (such as IDE plugins) to warn about filenames used by the standard library or by third-party libraries in the project. In Python 3.10 and up, the full list of standard library module names is also available as sys.stdlib_module_names
. This includes names that might not be present in the current Python installation (e.g. OS-specific components or things that are sometimes disabled or omitted, such as Tkinter).