How to fix "Attempted relative import in non-package" even with __init__.py

Question

I'm trying to follow PEP 328, with the following directory structure:

pkg/
  __init__.py
  components/
    core.py
    __init__.py
  tests/
    core_test.py
    __init__.py

In core_test.py I have the following import statement

from ..components.core import GameLoopEvents

However, when I run, I get the following error:

tests$ python core_test.py 
Traceback (most recent call last):
  File "core_test.py", line 3, in <module>
    from ..components.core import GameLoopEvents
ValueError: Attempted relative import in non-package

Searching around I found "relative path not working even with __init__.py" and "Import a module from a relative path" but they didn't help.

Is there anything I'm missing here?

Pruchno

I was also very confused by the various ways of structuring unittest projects, so I wrote this fairly exhaustive sample project that covers deep nesting of modules, relative and absolute imports (where the work and don't), and relative and absolute referencing from within a package, as well as single, double, and package-level import of classes. Helped clear things right up for me! – Glitter

I could not get your tests to work. Keep getting no module named myimports.foo when I run them. – Idler

@Idler I'm guessing the intended invocation is to cd into PyImports, and run python -m unittest tests.test_abs, for example. – Uranus

I agree with Gene. I wish there was a mechanism for debugging the importing process that was a little more helpful. In my case, I have two files in the same directory. I am trying to import one file into the other file. If I have an init.py file in that directory, I get a ValueError: Attempted relative import in non-package error. If I remove the init.py file, then I get an error no module named 'NAME' error. – Bezanson

In my case, I have two files in the same directory. I am trying to import one file into the other file. If I have an init.py file in that directory, I get a ValueError: Attempted relative import in non-package error. If I remove the init.py file, then I get an error no module named 'NAME' error. What is really frustrating is that I had this working, and then I shot myself in the foot by deleting the .bashrc file, which set the PYTHONPATH to something, and now it is not working. – Bezanson

@user1928764: These errors are indeed frustrating. I assume you meant "__init__.py file." – Tuff

Have a look at this: https://mcmap.net/q/14223/-relative-imports-for-the-billionth-time – Devisal

Answer 1

Yes. You're not using it as a package.

python -m pkg.tests.core_test

Egalitarian

A gotcha: Note that there is no '.py' at the end! – Jumbuck

I'm not either of the downvoters, but I feel this could use quite a bit more detail, given the popularity of this question and answer. Noting stuff like from what directory to execute the above shell command, the fact that you need __init__.pys all the way down, and the __package__-modifying trickery (described below by BrenBarn) needed to allow these imports for executable scripts (e.g. when using a shebang and doing ./my_script.py at the Unix shell) would all be useful. This whole issue was quite tricky for me to figure out or find concise and understandable documentation on. – Alda

There is a nicely elaborated recent answer below with a link to the docs as suggested by the upvoted comment. – Bavardage

Note: you need to be outside of the directory pkg at the point where you call this line from the CLI. Then, it should work as expected. If you are inside pkg and you call python -m tests.core_test, it will not work. At least it didn't for me. – Idler

This does not work if you run the script inside components dir for example – Eating

I coupled this with cding into the project's parent directory. cd /path/to/project && python -m project.module.submodule. You may want to add && cd ~ in the end if you are testing this out multiple times on local system. – Nureyev

When they designed the language, why didn't they just make "python pkg/test/core_test.py" do the same thing as "python -m pkg.test.core_test"? Why do they deliberately break relative imports whenever you run anything as a script? – Supereminent

@JonathanRay: Because having them do the same thing makes execution ambiguous; what sense does it make for python /usr/share/someapp/somescript.py to be usr.share.someapp.somescript? – Egalitarian

Seriously, can you explain whats going on in your answer? – Branum

@MarkAmery Almost lost my mind trying to grok how all this works, relative imports within a project with subdirectories with py files that have __init__.py files yet you keep getting the ValueError: Attempted relative import in non-package error. I would pay really good money for someone, somewhere, to finally explain in plain English how all of this works. – Brookes

Contrary to the answer for an FAQ in Python this answer gives no clue what is happening and from where to execute this command – Skat

for running on jupyter notebook %run <filename/path> -m – Narayan

I'm not sure how running it as a module helps @Pruchno 's import problem from within another script... – Chemotherapy

Maybe I don't really understand the cultural norms around the wiki-like features on SO, but it's notable that no one pointing out how Ignacio's answer could be improved actually improved it in-place. (Mind you, I agree 100% with the comments that an explanation would be extremely helpful.) – Supercolumnar

-m flag is used to run the file as a script. It indicates the module name and starts from the parent module and goes deeper in the directory to run the script. – Ethyne

What does the answer actually mean? – Saphead

@Supercolumnar Probably because no one actually knows how or why it works? That's the reason people want OP to add detail to the answer. If people already knew the details, they wouldn't need OP to add the detail themselves. – Duelist

@Duelist -- fair point. My perception is probably biased slightly by my observation people seem generally slow to view the questions as wiki-like. And I don't want to start riding roughshod over cultural norms. – Supercolumnar

@Supercolumnar part of the problem is that answers (and questions) accumulate reputation for the original author over time, which isn't shared by editors. Even if it were, there wouldn't be a tractable way to assess how much of the credit should go to each participant. There are some norms around authorial intent, but I honestly think people are mainly held back by a poor incentive structure. – Dagmardagna

this works but not when using pytest :( – Disenable

Answer 2

To elaborate on Ignacio Vazquez-Abrams's answer:

The Python import mechanism works relative to the __name__ of the current file. When you execute a file directly, it doesn't have its usual name, but has "__main__" as its name instead. So relative imports don't work.

You can, as Igancio suggested, execute it using the -m option. If you have a part of your package that is meant to be run as a script, you can also use the __package__ attribute to tell that file what name it's supposed to have in the package hierarchy.

See http://www.python.org/dev/peps/pep-0366/ for details.

Ostrander

Took a while for me to realize you can't run python -m core_test from within the tests subdirectory - it has to be from the parent, or you have to add the parent to the path. – Sloshy

So can I therefore use package to ensure executable script files are able to relative import from the rest of the system regardless of their level in the hierarchy? This would be very useful indeed. – Gehring

@DannyStaple: Not exactly. You can use __package__ to ensure executable script files can relatively import other modules from within the same package. There's no way to relatively import from "the whole system". I'm not even sure why you'd want to do this. – Ostrander

I mean if the __package__ symbol is set to "parent.child" then you'd be able to import "parent.other_child". Perhaps I didn't phrase it so well. – Gehring

@DannyStaple: Well, how it works is described in the linked documentation. If you have a script script.py in package pack.subpack, then setting it's __package__ to pack.subpack will let you do from ..module import something to import something from pack.module. Note that, as the documentation says, you still have to have the top-level package on the system path. This is already the way things work for imported modules. The only thing __package__ does is let you use that behavior for directly-executed scripts as well. – Ostrander

There is no sample code in the PEP-0366 linked documentation @BrenBarn. Where can I find sample code? – Agrology

@AlisonS: Sample code to do what? There is a small snippet in the PEP that explains how to set __package__. – Ostrander

But there is no context to where that snippet belongs @BrenBarn. Does it go in the file you're importing? In the file that is doing the import? Where in the relevant file does it go? – Agrology

@AlisonS: It says right there: "To allow relative imports when the module is executed directly, boilerplate similar to the following would be needed before the first relative import statement". I think that is pretty clear that you put it in the module you are executing directly, before that module attempts any relative imports. – Ostrander

I use __package__ in the script which is executed directly but Unfortunately, I get the the following error: "Parent module 'xxx' not loaded, cannot perform relative import" – Lorrettalorri

@AramKocharyan aside from hacking the path within the script, it should also work to set the PYTHONPATH environment variable appropriately. – Dagmardagna

Answer 3

It depends on how you want to launch your script.

If you want to launch your UnitTest from the command line in a classic way, that is:

python tests/core_test.py

Then, since in this case 'components' and 'tests' are siblings folders, you can import the relative module either using the insert or the append method of the sys.path module. Something like:

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from components.core import GameLoopEvents

Otherwise, you can launch your script with the '-m' argument (note that in this case, we are talking about a package, and thus you must not give the '.py' extension), that is:

python -m pkg.tests.core_test

In such a case, you can simply use the relative import as you were doing:

from ..components.core import GameLoopEvents

You can finally mix the two approaches, so that your script will work no matter how it is called. For example:

if __name__ == '__main__':
    if __package__ is None:
        import sys
        from os import path
        sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
        from components.core import GameLoopEvents
    else:
        from ..components.core import GameLoopEvents

Tare

what shall i do if i am trying to use the pdb for debugging? since you use python -m pdb myscript.py to launch the debugging session. – Tsushima

@dannynjust -- That's a good question since you can't have 2 main modules. Generally when debugging, I prefer to drop into the debugger manually at the first point where I want to start debugging. You can do that by inserting a import pdb; pdb.set_trace() into the code (inline). – Bitten

Is it better to use insert instead of append? That is, sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) – Royce

Using insert is a better match for the relative import semantics, where local package names take precedence over installed packages. Especially for tests, you usually want to test the local version, not the installed one (unless your test infrastructure installs the code under test, in which case relative imports are unneeded and you won't have this problem). – Antiquate

you should also mention that you can't be in the directory containing core_test when you run as a module (that would be too easy) – Eavesdrop

Answer 4

You can use import components.core directly if you append the current directory to sys.path:

if __name__ == '__main__' and __package__ is None:
    from os import sys, path
    sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))

Albuminoid

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) this will also work – Plaudit

from os import sys looks like cheating :) – Ammadas

@Plaudit And yours is better because of what? – Illaffected

@Piotr: It's might be considered better because it slightly shows more clearly what is being being appended to sys.path -- the parent of the directory the current file is in. – Tubing

@Tubing I think you are right :) – Illaffected

@Piotr: Ah, yes, I would have to agree that sys.path.append(path.dirname(path.dirname(__file__))) would be even more clear -- and with a comment, perfect. – Tubing

@flyingsheep: Agreed, I'd just use a regular import sys, os.path as path. – Tubing

FYI, to use this in an ipython notebook, I adapted this answer to: import os; os.sys.path.append(os.path.dirname(os.path.abspath('.'))). Then a straight import components.core works for me, importing from the notebook's parent directory as desired. – Ridglea

Great things for nested usage of dirname function. Haven't seen that before. – Necropolis

How does from os import sys work? os is a module outside the sys module. – Mathis

May I add another trick... I tried every advise that I could find on SO regarding a similar issue and finally found that I had some old .pyc files that were causing conflicts. So if anyone tries the above solution and it fails, check if you need to delete your .pyc files. – Duyne

doing sys.path.append is not cool to do because generating a standalone installer with pyinstall... fails miserably. I switched to file system links – Aspa

Answer 5

In core_test.py, do the following:

import sys
sys.path.append('../components')
from core import GameLoopEvents

Haitian

Answer 6

The issue is with your testing method.

You tried python core_test.py and then you will get this error:

ValueError: Attempted relative import in non-package

Reason: you are testing your packaging from a non-package source.

So test your module from the package source.

If this is your project structure,

pkg/
  __init__.py
  components/
    core.py
    __init__.py
  tests/
    core_test.py
    __init__.py

cd pkg

python -m tests.core_test # dont use .py

or from outside pkg/

python -m pkg.tests.core_test

single . if you want to import from a folder in the same directory. For each step back, add one more.

hi/
  hello.py
how.py

In how.py

from .hi import hello

in case if you want to import how from hello.py:

from .. import how

Emphasis

In the example from .. import how, how do you import a specific class / method from the 'how' file. when I do the equivalent of from ..how import foo then I get "attempted relative import beyond top-level package" – Lvov

@JamesHulse Does from .. import how work but the second statement not? I would have assumed that both statements won't work, if the folder that contains the hi folder does not contain an _ _ init _ _.py file. In other words, if there is only an init file in the hi folder, then the hi folder is the top level package and you cannot address beyond it. – Assuntaassur

Answer 7

If your use case is for running tests, and it seams that it is, then you can do the following. Instead of running your test script as python core_test.py use a testing framework such as pytest. Then on the command line you can enter

$$ py.test

That will run the tests in your directory. This gets around the issue of __name__ being __main__ that was pointed out by @BrenBarn. Next, put an empty __init__.py file into your test directory, this will make the test directory part of your package. Then you will be able to do

from ..components.core import GameLoopEvents

However, if you run your test script as a main program then things will fail once again. So just use the test runner. Maybe this also works with other test runners such as nosetests but i haven't checked it. Hope this helps.

Overpraise

Answer 8

My quick-fix is to add the directory to the path:

import sys
sys.path.insert(0, '../components/')

Cocoon

Your approach wont work in all cases because '../' part is resolved from directory from which you run your script(core_test.py). With your approach you are forced to cd to 'tests' before running the core_test.py scritp. – Monarch

Answer 9

As you have already marked everything as a module, there isn't any need to use the relative reference if you launch it as a Python module.

Instead of

from ..components.core import GameLoopEvents

simply

from pkg.components.core import GameLoopEvents

When you run from the parent of the package, use the following:

python -m pkg.tests.core_test

Shadoof

Nice! I was having trouble testing my package using this style of imports because python would pick up an older, installed, version of my package rather than the new version I was testing. That's when running as python pkg/tests/core_test.py or similar. But using -m solves this, seems to now prefer the local copy, I guess because it is considered a module now. Kind of makes me wonder when relative imports are ever a good idea though... – Apocalyptic

Answer 10

As Paolo said, we have two invocation methods:

1. python -m tests.core_test
2. python tests/core_test.py

One difference between them is the sys.path[0] string. Since the interpret will search sys.path when doing import, we can do with tests/core_test.py:

if __name__ == '__main__':
    import sys
    from pathlib import Path
    sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
    from components import core
    <other stuff>

And more after this, we can run core_test.py with other methods:

cd tests
python core_test.py
python -m core_test
...

Note, tested only on Python 3.6.

Goldman

Answer 11

I found out that adding an __all__= ['submodule', ...] to the init.py file and then using the from <CURRENT_MODULE> import * in the target works fine.

Fertilization

Answer 12

You can use from pkg.components.core import GameLoopEvents. For example, I use PyCharm.

The below is my project structure image. I just import from the root package, and then it works:

Pluralism

This did not work for me. Did you have to set the path in your configuration? – Cowherb

@MohammadMahjoub You need all the init files that he also has to make this work... Don't forget the one in cnn_scratch – Assuntaassur

Answer 13

This approach worked for me and is less cluttered than some solutions:

try:
  from ..components.core import GameLoopEvents
except ValueError:
  from components.core import GameLoopEvents

The parent directory is in my PYTHONPATH, and there are __init__.py files in the parent directory and this directory.

The above always worked in Python 2, but Python 3 sometimes hit an ImportError or ModuleNotFoundError (the latter is new in Python 3.6 and a subclass of ImportError), so the following tweak works for me in both Python 2 and Python 3:

try:
  from ..components.core import GameLoopEvents
except ( ValueError, ImportError):
  from components.core import GameLoopEvents

React

Answer 14

Try this

import components
from components import *

Infect

Answer 15

python <main module>.py does not work with relative import

The problem is relative import does not work when you run a __main__ module from the command line

python <main_module>.py

It is clearly stated in PEP 338.

The release of 2.5b1 showed a surprising (although obvious in retrospect) interaction between this PEP and PEP 328 - explicit relative imports don't work from a main module. This is due to the fact that relative imports rely on __name__ to determine the current module's position in the package hierarchy. In a main module, the value of __name__ is always '__main__', so explicit relative imports will always fail (as they only work for a module inside a package).

Cause

• Python Bug Tracker Issue1510172: Absolute/relative import not working?

The issue isn't actually unique to the -m switch. The problem is that relative imports are based on __name__, and in the main module, __name__ always has the value __main__. Hence, relative imports currently can't work properly from the main module of an application, because the main module doesn't know where it really fits in the Python module namespace (this is at least fixable in theory for the main modules executed through the -m switch, but directly executed files and the interactive interpreter are completely out of luck).

To understand further, see Relative imports in Python 3 for the detailed explanation and how to get it over.

Palatalized

Answer 16

Here is a workaround. Here's a bit of context. I wanted to test out one of the methods I've in a file. When I run it from within

if __name__ == "__main__":

it always complained of the relative imports. I tried to apply the above solutions, but failed to work, since there were many nested files, each with multiple imports.

Here's what I did. I just created a launcher, an external program that would import necessary methods and call them. Though, not a great solution, it works.

Heavensent

Answer 17

Here's one way which will piss off everyone, but it works pretty well. In tests, run:

ln -s ../components components

Then just import components like you normally would.

Bullish

This is a seriously undervalued gem! I can't understand why there are not enough up-votes to make this answer near the top! – Pythagoras

Answer 18

For me only this worked: I had to explicitly set the value of package to the parent directory, and add the parent directory to sys.path

from os import path
import sys
if __package__ is None:
    sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
    __package__= "myparent"

from .subdir import something # the . can now be resolved

I can now directly run my script with python myscript.py.

Sahaptin

Answer 19

If your project structure would look like this:

   project
     |
     | --- module1
     |      |
     |      file1.py
     |
     |-----module2
     |     |
     |     file2.py

and you are going import file1.py from within file2.py, you can do this in file2.py:

import sys
sys.path.append('.')

import file2

I still don't know why and how, but it worked for me.

Mclaurin

Are sure it is not import file1? In my case I use sys.path.append("../") and then import module1.file1. – Snort

Answer 20

This is very confusing and if you are using IDE like PyCharm, it's little more confusing.

What worked for me:

1. Make PyCharm project settings (if you are running Python from a venv or from Python directory)

2. There is nothing wrong with the way you defined. Sometime it works with:

   from folder1.file1 import class
   

   If it does not work, use:

   import folder1.file1
   

3. Your environment variable should be correctly mentioned in system or provide it in your command-line argument.

Rubino

Answer 21

Let's say you DON'T want to do any of these:

• run with python -m pkg.tests.core_test
• edit __package__
• add if check in __main__
• edit sys.path by hand
• restructure the project

Here is what you can do instead:

$ pip install importmonkey [github] [pip] [docs]

├─ src
│   └─ project
│       ├─ __init__.py
│       └─ module.py
└─ test
    └─ test.py

# In test.py

from importmonkey import add_path
add_path("../src")  # relative to current __file__
import project

Subjunction

Answer 22

Because your code contains if __name__ == "__main__", which doesn't be imported as a package, you'd better use sys.path.append() to solve the problem.

Poundal

I don't think having if __name__ == "__main__" in your file makes a difference to anything related to importing. – Heida