how to exclude "tests" folder from the wheel of a pyproject.toml managed lib?

Question

I try my best to move from a setup.py managed lib to a pure pyproject.toml one. I have the following folder structure:

tests
└── <files>
docs
└── <files>
sepal_ui
└── <files>
pyproject.toml

and in my pyproject.toml the following setup for file and packages discovery:

[build-system]
requires = ["setuptools>=61.2", "wheel"]

[tool.setuptools]
include-package-data = false

[tool.setuptools.packages.find]
include = ["sepal_ui*"]
exclude = ["docs*", "tests*"]

and in the produce wheel, I get the following:

tests
└── <files>
docs
└── <files>
sepal_ui
└── <files>
sepal_ui.egg-info
└── top-level.txt

looking at the top-level.txt, I see that only sepal_ui is included so my question is simple why do the extra "docs" and "tests" folder are still included even if they are not used? how to get rid of them ?

PS: I'm aware of the MANIFEST.in solution that I will accept if it's really the only one but I found it redundant to specify in 2 files.

Tripos

that doesn't change anything but that's expected as both "docs" and "tests" are outside "sepal_ui" folder/package – Tripos

Ooops, did not get the structure right. Anyway, I tried to reproduce it and I am getting a correct .whl file (checked with: unzip -l <WHEELFILE>). No docs, no tests. – Much

I'm moving toward a solution, I had my egg-info/ and build/ folder polluting my local experiments. now the docs files are gone but the tests are very sticky. here is the PR if you want to have a look: github.com/12rambau/sepal_ui/pull/742 – Tripos

Answer 1

Fun fact, it was working from the start....

Small debugging workflow for the next person that does not want to spend hours for nothing.

configuration of the pyproject.toml

the following configuration is the minimal to remove files from a docs/ and tests/ folders that are at the root of the repository. If you disseminated your tests in each modules, consider adding *.tests*:

[build-system]
requires = ["setuptools>=61.2", "wheel"]

[tool.setuptools]
include-package-data = false

[tool.setuptools.packages.find]
include = ["sepal_ui*"]
exclude = ["docs*", "tests*"]

Clean environment

That was my mistake. Python is cahing information in the build/ and egg-info folder. setuptools will use the lest of files stored in egg-info/SOURCE.txt so first step: get rid of previous build files.

Then simply run:

python -m build

check both wheel and tar.gz

From the start I was checking only tar.gz thinking (naively) that .whl and .tar.gz were the same. It's not while the tests folder remains in the tar.gz file, it's absent from the wheel.

Tripos

also see stackoverflow.com/a/59686298 – Tubule

related question about files in .tar.gz but not in wheel: https://mcmap.net/q/829962/-folders-included-in-the-tar-gz-not-in-the-wheel-setuptools-build – Tubule

and possibly related issue: github.com/pypa/setuptools/issues/3817 – Tubule

As per in #8557496, because I had a __init__.py in my tests/ and a non-test module, I needed to add recursive-exclude tests * to MANIFEST.in for anything under tests not be included. – Lawler

all my packages have a __init__.py file to make them compatible with a pytest plugin I use and I never needed to add anything in the MANIFEST.in. – Tripos

Answer 2

This has worked for me for hatchling:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build]
exclude = [
  "/.*",
  "/docs",
  "/tests",
]

Drayage

Thanks for this. This helped me to do exactly what I needed to do until our "packaging" standards change to keep only source files one folder. For now, we have everything under then main package folder, so this helped to isolate unneeded venv, test, docs, etc. – Leman

Answer 3

For Flit:

[build-system]
requires = ["flit_core >=2,<4"]
build-backend = "flit_core.buildapi"

[tool.flit.sdist]
include = ["doc/"]
exclude = ["doc/*.html"]

Archaeozoic

Answer 4

According to the docs, you still need MANIFEST.in.

The package tree

project_root_directory
├── pyproject.toml
└── src
    └── mypkg
        ├── __init__.py
        ├── data1.rst
        ├── data2.rst
        ├── data1.txt
        └── data2.txt

pyproject.toml

[tool.setuptools]
# ...
# By default, include-package-data is true in pyproject.toml, so you do
# NOT have to specify this line. Either remove this or ensure it's not 
# set to false.
include-package-data = true

[tool.setuptools.packages.find]
where = ["src"]

MANIFEST.in

include src/mypkg/*.txt
include src/mypkg/*.rst

Then (and only then) all the .txt and .rst files will be automatically installed with your package.

Use prune and/or recursive-exclude in MANIFEST.in to control what should not be included.

Biped

I try not to use more files than necessary and according to the doc you just linked explicitely shows that you can do everything from the pyproject.toml file using package-data argument. Note that this question covers another topic as I was specifically using include-package-data = false. – Tripos