import local package during poetry run
Asked Answered
R

3

22

I just transitioned from pipenv to poetry and I'm having trouble importing a package from a local package I'm developing in a few of my scripts. To make this more concrete, my project looks something like:

pyproject.toml
poetry.lock
bin/
  myscript.py
mypackage/
  __init__.py
  lots_of_stuff.py

Within myscript.py, I import mypackage. But when I poetry run bin/myscript.py I get a ModuleNotFoundError because the PYTHONPATH does not include the root of this project. With pipenv, I could solve that by specifying PYTHONPATH=/path/to/project/root in a .env file, which would be automatically loaded at runtime. What is the right way to import local packages with poetry?

I ran across this piece on using environment variables but export POETRY_PYTHONPATH=/path/to/roject/root doesn't seem to help.

Rabaul answered 4/3, 2021 at 12:15 Comment(1)
Maybe you should install mypackage using poetry. From what I understand this adds the path to the root of your project to the PYTHONPATH. Have you tried running poetry install and making sure that mypackage gets installed inside the virtual environment of your project and running the script again?Chordate
R
13

After quite a bit more googling, I stumbled on the packages attribute within the tool.poetry section for pyproject.toml files. To include local packages in distribution, you can specify

# pyproject.toml

[tool.poetry]
# ...
packages = [
    { include = "mypackage" },
]

Now these packages are installed in editable mode :)

Rabaul answered 5/3, 2021 at 12:46 Comment(3)
This is only necessary if the name of the folder with the package data differs from what is written [tool.poetry] section for the keyword name.Sammy
Good to know. Thanks @Sammy for the context. And thanks especially for making poetry, which is 10x better than pipenv IMHORabaul
Thanks! This works after I included my folder with all code { include = "src" }. I wonder if I am missing some env var as poetry run h1-script only works when I am in the src folder and not the project root folder..Pelvic
T
29

Adding a local package (in development) to another project can be done as:

poetry add ./my-package/
poetry add ../my-package/dist/my-package-0.1.0.tar.gz
poetry add ../my-package/dist/my_package-0.1.0.whl

If you want the dependency to be installed in editable mode you can specify it in the pyproject.toml file. It means that changes in the local directory will be reflected directly in environment.

[tool.poetry.dependencies]
my-package = {path = "../my/path", develop = true}

With current preview release (1.2.0a) a command line option was introduced, to avoid above manually steps:

poetry add --editable /path/to/package

Another ways of adding packages can be found on poetry add page

If the above doesn't work, you can take a look over additional steps detailed in this discussion

Traitorous answered 8/11, 2021 at 10:34 Comment(3)
Today, poetry available version is 1.1.11, 1.2.0 still in alphaTraitorous
however this pollutes pyproject.toml, which is under source control. This can cause troublesIndivertible
@Indivertible I agree, and have opened an issue related to that.Ezekielezell
R
13

After quite a bit more googling, I stumbled on the packages attribute within the tool.poetry section for pyproject.toml files. To include local packages in distribution, you can specify

# pyproject.toml

[tool.poetry]
# ...
packages = [
    { include = "mypackage" },
]

Now these packages are installed in editable mode :)

Rabaul answered 5/3, 2021 at 12:46 Comment(3)
This is only necessary if the name of the folder with the package data differs from what is written [tool.poetry] section for the keyword name.Sammy
Good to know. Thanks @Sammy for the context. And thanks especially for making poetry, which is 10x better than pipenv IMHORabaul
Thanks! This works after I included my folder with all code { include = "src" }. I wonder if I am missing some env var as poetry run h1-script only works when I am in the src folder and not the project root folder..Pelvic
H
0

As of Poetry version 1.8.3, neither @dino's answer nor @mmsilviu's answer worked for me. The reason for this was that I kept getting the error explained in this issue. This comment hints that Poetry expects packages to be in a specific format, which is what fixed the problem for me in the end.

I've split my answer up into a "Short version" section, where I briefly summarize the directory structure you need, and a "Long version" section, where I explain how to build up this directory structure using Poetry.

Short version

To summarize, your complete directory structure should be as follows:

src/
|-- pyproject.toml
|-- scripts/
|-- |-- main.py
|-- my_package/
|-- |-- my_package/
|-- |-- |-- __init__.py
|-- |-- |-- hello_world.py
|-- |-- tests/
|-- |-- |-- __init__.py
|-- |-- pyproject.toml
|-- |-- README.md

where the contents of each file are as follows (I included a comment at the top of each code snippet to indicate the file I'm referring to):

# src/pyproject.toml
[tool.poetry]
name = "src"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
package-mode = false

[tool.poetry.dependencies]
python = "^3.10"
my-package = { path = "my_package", develop = true }


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
# src/scripts/main.py
from my_package.hello_world import hello_world

hello_world()
# src/my_package/my_package/__init__.py
from .hello_world import hello_world
# src/my_package/my_package/hello_world.py
def hello_world():
    print("Hello World!")
# src/my_package/tests/__init__.py
# this file is empty
# src/my_package/pyproject.toml
[tool.poetry]
name = "my-package"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[comment]: # (src/my_package/README.md)
[comment]: # (this file is empty but can be populated as needed.)

Long version

More specifically, suppose you start with the following directory structure:

src/
|-- scripts/
|-- |-- main.py

where the content of the scripts/main.py file is

# scripts/main.py
from my_package.hello_world import hello_world

hello_world()

So far, the my_package package does not yet exist. We will use Poetry to first create a top-level pyproject.toml file in the src directory, and then use Poetry to create the my_package package so that it has the correct structure. We do so using the following steps:

  1. While in the src directory, run poetry init. Press "Enter" for each option and enter "no" when asked to specify dependencies interactively. This will create a pyproject.toml file under the src directory.

  2. Open the pyproject.toml file that you just created and add the line package-mode = false under the section [tool.poetry]. Also, add the line my-package = { path = "my_package", develop = true } under the section [tool.poetry.dependencies].

  3. While in the src directory, run poetry new my_package. This will create the directory my_package and the desired sub-directories.

  4. Under the directory src/my_package/my_package, create the file hello_world.py and put in it the following code:

    # src/my_package/my_package/hello_world.py
    def hello_world():
        print("Hello World!")
    
  5. Under the directory src/my_package/my_package, create the file __init__.py and put in it the following code:

    # src/my_package/my_package/__init__.py
    from .hello_world import hello_world
    
  6. You should now be ready to run poetry install. While in the src directory, run poetry install, and the my_package package should install successfully.

  7. Finally, while in the src directory, run poetry shell, and then run python scripts/main.py. You should then see Hello World! at the output, which indicates that the scripts/main.py file ran successfully.

Hookup answered 7/6 at 20:1 Comment(1)
Interesting. My solution above still works for me on the latest version of poetry and I don't need to have the double my_package/my_package directory.Rabaul

© 2022 - 2024 — McMap. All rights reserved.