beyond top level package error in relative import [duplicate]

Question

It seems there are already quite some questions here about relative import in python 3, but after going through many of them I still didn't find the answer for my issue. so here is the question.

I have a package shown below

package/
   __init__.py
   A/
      __init__.py
      foo.py
   test_A/
      __init__.py
      test.py

and I have a single line in test.py:

from ..A import foo

now, I am in the folder of package, and I run

python -m test_A.test

I got message

"ValueError: attempted relative import beyond top-level package"

but if I am in the parent folder of package, e.g., I run:

cd ..
python -m package.test_A.test

everything is fine.

Now my question is: when I am in the folder of package, and I run the module inside the test_A sub-package as test_A.test, based on my understanding, ..A goes up only one level, which is still within the package folder, why it gives message saying beyond top-level package. What is exactly the reason that causes this error message?

Alfreda

I have a thought here, so when run test_A.test as module, ‘..' goes above test_A, which is already the highest level of the import test_A.test, I think the package level is not the directory level, but how many levels you import the package. – Alfreda

I promise you will understand everything about relative import after watching this answer https://mcmap.net/q/14223/-relative-imports-for-the-billionth-time. – Frow

Is there a way to avoid doing relative imports? Such as the way PyDev in Eclipse sees all packages within <PydevProject>/src? – Phiphenomenon

Does your working dir also have an init.py? – Henryhenryetta

See also: Relative imports in Python 3 – Forte

Answer 1

EDIT: There are better/more coherent answers to this question in other questions:

• Sibling package imports
• Relative imports for the billionth time

---

Why doesn't it work? It's because python doesn't record where a package was loaded from. So when you do python -m test_A.test, it basically just discards the knowledge that test_A.test is actually stored in package (i.e. package is not considered a package). Attempting from ..A import foo is trying to access information it doesn't have any more (i.e. sibling directories of a loaded location). It's conceptually similar to allowing from ..os import path in a file in math. This would be bad because you want the packages to be distinct. If they need to use something from another package, then they should refer to them globally with from os import path and let python work out where that is with $PATH and $PYTHONPATH.

When you use python -m package.test_A.test, then using from ..A import foo resolves just fine because it kept track of what's in package and you're just accessing a child directory of a loaded location.

Why doesn't python consider the current working directory to be a package? NO CLUE, but gosh it would be useful.

Syllabize

I've edited my answer to refer to a better answer to a question that amounts to the same thing. There is only workarounds. The only thing that I've actually seen work is what the OP has done, which is use the -m flag and run from the directory above. – Syllabize

It should be noted that this answer, from the link given by Multihunter, does not involve the sys.path hack, but the use of setuptools, which is much more interesting in my opinion. – Triturable

so the ".." in relative import is not as we usually think of ".." in the context of os? In os, .. simply mean step out the current dir and move to the parent dir, which is not the case in python's relative import. As you mentioned, the interpret discard the package information of current working directory. To me, as a developer who gets used to the os path concepts, it is little unintuitive :-( – Names

"Why doesn't python consider the current working directory to be a package? NO CLUE, but gosh it would be useful." Rather than that, I wish it were possible to specify, in conjunction with the -m option, a path to the package root for the specified package. – Caracul

How about the people who write python just fix this nonsense? And we can get back to coding again. – Tartarous

@Tartarous they know this is a problem but all the proposed solutions so far have some weird consequences and they tend to not accept changes when that's the case. – Cicero

Answer 2

import sys
sys.path.append("..") # Adds higher directory to python modules path.

Try this. Worked for me.

Dreiser

Umm...how woudl this work? Every single test file would have this? – Maighdlin

The problem here is if, e.g., A/bar.py exists and in foo.py you do from .bar import X. – Hagiocracy

I had to remove the .. from "from ..A import..." after adding the sys.path.append("..") – Laaland

If script is executed from outside the directory it exists, this wouldn't work. Instead, you have to tweak this answer to specify absolute path of the said script. – Leisured

this is the best, least complicated option – Hockenberry

@AlexR, I would rather say this is the easiest short-term solution, but certainly not the best. – Transfix

But this gets nasty if you then perform isinstance checks on for instance classes defined in python files in the subdir, as they then have i prefix... – Yocum

Bingo. Another option is to permanently add it to the path file. – Soricine

I don't think this is a good solution. I explain why this happens and provide solutions here stackoverflow.com/questions/30669474/… – Nitrometer

This does not generalize well, and is brittle (you end up with a lot of entries in the sys.path, and it isn't clear what order they'll be in, which makes it difficult to figure out which folders will be searched in what order). It causes a lot of problems if you want to have modules with the same name in different sub-packages. Like the Zen of Python says, Namespaces are a honking great idea, and this stomps on that. – Caracul

This (sys.path.append("..")) only works if you are inside the directory of the script you are about to run, when you run it. Otherwise, you need to do this instead: assume you are running script mydir/mypackage/myscript.py, then you'd need to do sys.path.append("mydir") in order to give access to mypackage. However, if you are inside dir mypackage when you run the script, then you need to do sys.path.append("..") to give access to mypackage. The goal is to give myscript.py access to mypackage, based relative to the directory you are in when you run myscript.py! – Forte

Answer 3

Assumption:
If you are in the package directory, A and test_A are separate packages.

Conclusion:
..A imports are only allowed within a package.

Further notes:
Making the relative imports only available within packages is useful if you want to force that packages can be placed on any path located on sys.path.

EDIT:

Am I the only one who thinks that this is insane!? Why in the world is the current working directory not considered to be a package? – Multihunter

The current working directory is usually located in sys.path. So, all files there are importable. This is behavior since Python 2 when packages did not yet exist. Making the running directory a package would allow imports of modules as "import .A" and as "import A" which then would be two different modules. Maybe this is an inconsistency to consider.

Iconostasis

Am I the only one who thinks that this is insane!? Why in the world is the running directory not considered to be a package? – Syllabize

Not only is that insane, this is unhelpful...so how do you run tests then? Clearly the thing the OP was asking and why I'm sure many people are here as well. – Maighdlin

The running directory is usually located in sys.path. So, all files there are importable. This is behavior since Python 2 when packages did not yet exist. - edited answer. – Iconostasis

I don't follow the inconsistency. The behaviour of python -m package.test_A.test seems to do what is wanted, and my argument is that that should be the default. So, can you give me an example of this inconsistency? – Syllabize

I'm actually thinking, is there a feature request for this? This is indeed insane. C/C++ style #include would be very useful! – Nikko

"Why in the world is the running directory not considered to be a package?" First off: if I run python ../sibling/script.py, should Python assume . to be a package root? ../sibling? Both? If both, in what order would they be chosen? Second: the entire point of packages is that they represent a coherent bundle of installable code; you aren't supposed to go prowling around directories like that, just like you don't normally go prowling around C:\Program Files or /usr/local/bin. – Caracul

@NicholasHumphrey "C/C++ style #include" is a literal copy and paste of source code, prior to static compilation. It bears no resemblance to a proper import system whatsoever. If you simply mean that it lets you specify a file path directly, you can do that just fine in Python (using the importlib standard library module, which is a wrapper around the __import__ builtin function). In fact, Python gives you a wide variety of hooks into the import mechanism. #include is utterly primitive in comparison. – Caracul

@Syllabize slamming my head against the wall tryna figure this out. I have no idea how this is acceptable lol – Bedeck

Answer 4

None of these solutions worked for me in 3.6, with a folder structure like:

package1/
    subpackage1/
        module1.py
package2/
    subpackage2/
        module2.py

My goal was to import from module1 into module2. What finally worked for me was, oddly enough:

import sys
sys.path.append(".")

Note the single dot as opposed to the two-dot solutions mentioned so far.

---

Edit: The following helped clarify this for me:

import os
print (os.getcwd())

In my case, the working directory was (unexpectedly) the root of the project.

Hauberk

it's working locally but not working on aws ec2 instance, does it make any sense? – Binturong

This worked for me as well--in my case the working directory was likewise the project root. I was using a run shortcut from a programming editor (TextMate) – Boo

@Binturong Same! Works locally on my mac but does not work on ec2, then I realized that I was running the command in a subdir on ec2 and running it at root locally. Once I run it from root on ec2 it worked. – Climacteric

sys.path.append(".") worked because you are calling it in parent directory, note that . always represent the directory where you run python command in. – Forgiveness

This is called "Programming by coincidence" and it's absolutely awful. Don't do stuff you don't understand just because the code is working. It's absolutelty terrible this answer has so many votes. – Level

Printing the os.getcwd() got me the idea to solve my issue! Thank you! – Guinn

no idea why, but this worked for me – Frodeen

"In my case, the working directory was (unexpectedly) the root of the project." What did you expect to be the working directory instead? Why? – Caracul

@HenryHenrinson It's absolutely helpful. Finding a solution even by happenstance allows those of us who have the time/interest/abilities to dig deeper to do so at our leisure . Especially for a mostly arbitrary/ often illogical language like python. – Sydel

Last edit is what made me understand it as well. – Banner

Answer 5

This is very tricky in Python.

I'll first comment on why you're having that problem and then I will mention two possible solutions.

What's going on?

You must take this paragraph from the Python documentation into consideration:

Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.

And also the following from PEP 328:

Relative imports use a module's name attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to 'main') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.

Relative imports work from the filename (__name__ attribute), which can take two values:

1. It's the filename, preceded by the folder strucutre, separated by dots. For eg: package.test_A.test Here Python knows the parent directories: before test comes test_A and then package. So you can use the dot notation for relative import.

#  package.test_A/test.py
from ..A import foo

You can then have like a root file in the root directory which calls test.py:

#  root.py
from package.test_A import test

2. When you run the module (test.py) directly, it becomes the entry point to the program , so __name__ == __main__. The filename has no indication of the directory structure, so Python doesn't know how to go up in the directory. For Python, test.py becomes the top-level script, there is nothing above it. That's why you cannot use relative import.

---

Possible Solutions

A) One way to solve this is to have a root file (in the root directory) which calls the modules/packages, like this:

• root.py imports test.py. (entry point, __name__ == __main__).
• test.py (relative) imports foo.py.
• foo.py says the module has been imported.

The output is:

package.A.foo has been imported
Module's name is:  package.test_A.test

B) If you want to execute the code as a module and not as a top-level script, you can try this from the command line:

python -m package.test_A.test

Any suggestions are welcomed.

You should also check: Relative imports for the billionth time , specially BrenBarn's answer.

Nitrometer

Answer 6

from package.A import foo

I think it's clearer than

import sys
sys.path.append("..")

Rhodian

it's more readable for sure but still needs sys.path.append(".."). tested on python 3.6 – If

Same as older answers – Epiboly

Switching to absolute imports like this does not solve the problem. It just causes a different type of ImportError. – Caracul

This does not answer the question. – Sydel

Answer 7

As the most popular answer suggests, basically its because your PYTHONPATH or sys.path includes . but not your path to your package. And the relative import is relative to your current working directory, not the file where the import happens; oddly.

You could fix this by first changing your relative import to absolute and then either starting it with:

PYTHONPATH=/path/to/package python -m test_A.test

OR forcing the python path when called this way, because:

With python -m test_A.test you're executing test_A/test.py with __name__ == '__main__' and __file__ == '/absolute/path/to/test_A/test.py'

That means that in test.py you could use your absolute import semi-protected in the main case condition and also do some one-time Python path manipulation:

from os import path
…
def main():
…
if __name__ == '__main__':
    import sys
    sys.path.append(path.join(path.dirname(__file__), '..'))
    from A import foo

    exit(main())

Acquainted

"And the relative import is relative to your current working directory, not the file where the import happens" This is incorrect. The relative import is relative to where the import happens - that's why the error message is as OP describes. The top-level package is test_A in the error case, and the relative import tries to traverse into the parent package directory, starting at the file, not the current working directory. – Caracul

Answer 8

This is actually a lot simpler than what other answers make it out to be.

TL;DR: Import A directly instead of attempting a relative import.

The current working directory is not a package, unless you import the folder package from a different folder. So the behavior of your package will work fine if you intend it to be imported by other applications. What's not working is the tests...

Without changing anything in your directory structure, all that needs to be changed is how test.py imports foo.py.

from A import foo

Now running python -m test_A.test from the package directory will run without an ImportError.

Why does that work?

Your current working directory is not a package, but it is added to the path. Therefore you can import folder A and its contents directly. It is the same reason you can import any other package that you have installed... they're all included in your path.

Dody

This only works incidentally, for the specific case of trying to start from package/ (i.e. just inside the intended package root, rather than just outside). – Caracul

@Frodeen This works intentionally. These are test files, and the goal is to be able to run them while developing the package (i.e., the root directory is intentionally the package directory). If the root directory is not package and you are importing package as a library, your intent is likely not to run the tests. – Dody

Answer 9

Edit: 2020-05-08: Is seems the website I quoted is no longer controlled by the person who wrote the advice, so I'm removing the link to the site. Thanks for letting me know baxx.

---

If someone's still struggling a bit after the great answers already provided, I found advice on a website that no longer is available.

Essential quote from the site I mentioned:

"The same can be specified programmatically in this way:

import sys

sys.path.append('..')

Of course the code above must be written before the other import statement.

It's pretty obvious that it has to be this way, thinking on it after the fact. I was trying to use the sys.path.append('..') in my tests, but ran into the issue posted by OP. By adding the import and sys.path defintion before my other imports, I was able to solve the problem.

Mingy

Answer 10

Just remove .. in test.py For me pytest works fine with that
Example:

from A import foo

Valuation

While the answer is correct, the credit goes to Chris Collett answer. It appeared a year before this one. Also his answer explains things properly (why it does work). – Sememe

Answer 11

if you have an __init__.py in an upper folder, you can initialize the import as import file/path as alias in that init file. Then you can use it on lower scripts as:

import alias

Ballplayer

__init__.py has no bearing on the problem whatsoever. – Caracul

Answer 12

In my case, I had to change to this: Solution 1(more better which depend on current py file path. Easy to deploy) Use pathlib.Path.parents make code cleaner

import sys
import os
import pathlib
target_path = pathlib.Path(os.path.abspath(__file__)).parents[3]
sys.path.append(target_path)
from utils import MultiFileAllowed

Solution 2

import sys
import os
sys.path.append(os.getcwd())
from utils import MultiFileAllowed

Coliseum

Answer 13

In my humble opinion, I understand this question in this way:

[CASE 1] When you start an absolute-import like

python -m test_A.test

or

import test_A.test

or

from test_A import test

you're actually setting the import-anchor to be test_A, in other word, top-level package is test_A . So, when we have test.py do from ..A import xxx, you are escaping from the anchor, and Python does not allow this.

[CASE 2] When you do

python -m package.test_A.test

or

from package.test_A import test

your anchor becomes package, so package/test_A/test.py doing from ..A import xxx does not escape the anchor(still inside package folder), and Python happily accepts this.

In short:

• Absolute-import changes current anchor (=redefines what is the top-level package);
• Relative-import does not change the anchor but confines to it.

Furthermore, we can use full-qualified module name(FQMN) to inspect this problem.

Check FQMN in each case:

• [CASE2] test.__name__ = package.test_A.test
• [CASE1] test.__name__ = test_A.test

So, for CASE2, an from .. import xxx will result in a new module with FQMN=package.xxx, which is acceptable.

While for CASE1, the .. from within from .. import xxx will jump out of the starting node(anchor) of test_A, and this is NOT allowed by Python.

[2022-07-19] I think this "relative-import" limitation is quite an ugly design, totally against (one of) Python's motto "Simple is better than complex".

Jaimie

This is way more complicated than it needs to be. So much for Zen of Python. – Dissociable

Answer 14

Not sure in python 2.x but in python 3.6, assuming you are trying to run the whole suite, you just have to use -t

-t, --top-level-directory directory Top level directory of project (defaults to start directory)

So, on a structure like

project_root
  |
  |----- my_module
  |          \
  |           \_____ my_class.py
  |
  \ tests
      \___ test_my_func.py

One could for example use:

python3 unittest discover -s /full_path/project_root/tests -t /full_path/project_root/

And still import the my_module.my_class without major dramas.

Leslie

Answer 15

Having

package/
   __init__.py
   A/
      __init__.py
      foo.py
   test_A/
      __init__.py
      test.py

in A/__init__.py import foo:

from .foo import foo

when importing A/ from test_A/

import sys, os
sys.path.append(os.path.abspath('../A'))
# then import foo
import foo

Depredate