Python: How to make Poetry include a package/module that's not on a subpath?

Question

I have, in a single repo, two Python projects that both depend on a shared utility package. My goal is to package each of the two projects in a software distribution package (i.e. a .tzr.gz file)

I am currently getting this done using setuptools and setup.py files and having a hard time of it. I would much rather use Poetry to manage and package each of the two projects separately.

Please consider this "minimal repro" of my problem:

repo
    project1/ 
        __init__.py
        main_module.py
        pyproject.toml
    project2/ 
        __init__.py
        main_module.py
        pyproject.toml
    util/
        __init__.py
        util_module.py

I tried to get Poetry to include the util package when building project1 by modifying its project.toml this way:

[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = [""]
packages = [
    { include = "../util/*.py" }
]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

When I run poetry build I get this error:

Building project1 (0.1.0)
  - Building sdist

  ValueError

  'C:\\repo\\util\\__init__.py' is not in the subpath of 'C:\\repo\\project1' OR one path is relative and the other is absolute.

  at ~\.pyenv\pyenv-win\versions\3.9.6\lib\pathlib.py:929 in relative_to
       925│         n = len(to_abs_parts)
       926│         cf = self._flavour.casefold_parts
       927│         if (root or drv) if n == 0 else cf(abs_parts[:n]) != cf(to_abs_parts):
       928│             formatted = self._format_parsed_parts(to_drv, to_root, to_parts)
    →  929│             raise ValueError("{!r} is not in the subpath of {!r}"
       930│                     " OR one path is relative and the other is absolute."
       931│                              .format(str(self), str(formatted)))
       932│         return self._from_parsed_parts('', root if n == 1 else '',
       933│                                        abs_parts[n:])

Doesn't poetry support my use-case? If not, what am I missing?

Alternatively, please suggest another approach to package my two projects separately, but both packages must include the shared util package.

Right

Answer 1

This seems to be a issue with poetry. Check this out https://github.com/python-poetry/poetry/issues/5621.

Strict

Indeed the issue indicates that my use case is not currently supported in Poetry. In case someone finds this useful: I was able to get it done by switching to hatch as my build system and using its "force-include" feature. – Right

Hi @urig, Because of this issue, I am also evaluating hatch. I installed hatch but I dont see poetry add numpy equivalent in hatch. Do you know how to install packages? – Hexane

github.com/python-poetry/poetry-core/pull/… – Zenas

Answer 2

Util itself should be built as another python package (note the extra pyproject.toml)

repo
    project1/ 
        ...
    project2/ 
        ...
    util/
        ...
        pyproject.toml

Then add it as you normally would with any other package:

poetry add ../util

Which should add it to pyproject.toml:

[tool.poetry.dependencies]
python = "^3.9"
util = { path = "../util" }

Alphonsa