setuptools does not include any subpackages I specify

Question

I'm trying to create an installer for a CLI tool I'm developing with Click. The problem is that for some reason submodules aren't included in the installer, which results in the following error:

from modules.commands import modules
ModuleNotFoundError: No module named 'modules'

This is my directory structure:

.
├── LICENSE
├── README.md
├── setup.py
├── src
│   ├── app_config
│   │   ├── __init__.py
│   │   └── configuration.py
│   └── commands
│       ├── __init__.py
│       ├── config
│       │   ├── __init__.py
│       │   └── commands.py
│       ├── modules
│       │   ├── __init__.py
│       │   └── commands.py
│       └── cli.py
└── tests

cli.py references commands created in config and modules. When I'm running cli.py directly, it works fine. However, no matter what pattern I try for setup.py to include packages, the submodules in commands are not included.

This is the code for setup.py:

setup(
    name='cli',
    version='0.1.0',
    packages=find_packages(include=['src', 'src.*']),
    install_requires=[
        'Click'
    ],
    entry_points={
        'console_scripts': [
            'cli = src.commands.cli:cli'
        ]
    }
)

I've checked the find_packages-implementation, but I can't see anything wrong with the way I've specified what is written above. I've also tried hardcoding all packages, but that didn't work either. I've also tried with src/*, as maybe it uses full filepaths, but that didn't work either.

Edit: More Troubleshooting

Thanks to this question, I tried running python -c "from setuptools import setup, find_packages; print(find_packages('src'))". According to the output, it finds all the submodules:

['app_config', 'commands', 'commands.config', 'commands.modules']

However, when I update my setup to include the same value, it fails.

setup.py:

setup(
    name='cli',
    version='0.1.0',
    packages=find_packages('src'),
    install_requires=[
        'Click'
    ],
    entry_points={
        'console_scripts': [
            'cli = src.commands.cli:cli'
        ]
    }
)

Output:

  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
...
      error: package directory 'app_config' does not exist
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

Thither

I had a problem like this when I hardcoded package names in setup.py. I needed to add the subpackages to to packages in setup.py – Federicofedirko

Answer 1

you are probably missing MANIFEST.in file:

# https://packaging.python.org/guides/using-manifest-in/
graft src
global-exclude __pycache__
global-exclude *.py[cod]

and since you rely on setuptools as tool to build and distribute your package, it's recommended to adopt the last best practises, src layout (the case) and move the config to pyproject.toml and setup.cfg files:

pyproject.toml

[build-system]
# https://setuptools.readthedocs.io/en/latest/setuptools.html#setup-cfg-only-projects
requires = [
  "setuptools >= 40.9.0",
  "wheel"
]
build-backend = "setuptools.build_meta"

setup.py (optional)

from setuptools import setup

setup()

setup.cfg

# https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
[metadata]
name = [..]
version = [..]
description = [..]
long_description = file: README.rst
long_description_content_type = text/x-rst
author = [..]
author_email = [..]
# maintainer =
# maintainer_email =
license = [..]
license_file = LICENSE
# license_files =
url = [..]
download_url = [..]
project_urls =
    Documentation = [..]
    Issue Tracker = [..]
    Source Code = [..]
keywords = [..]
sclassifiers =
    [..]
platforms = any

[options]
python_requires = >=3.6
install_requires =
    Click
packages = find_namespace:
package_dir =
    = src
include_package_data = True
zip_safe = False

[options.packages.find]
where = src

[options.entry_points]
console_scripts =
    cli = commands.cli:cli  # Update: change `src.commands.cli:cli` to
                            # `commands.cli:cli` as per the comment below

[options.extras_require]
    [..]

if you are on windows and you want to build and publish your package

pip install build wheel twine
py -m build -n  # don't forget '-n' flage to force using your project venv
pip install -e .  # for editable mode 

# then, if everything is ok then pulish it on pypi
py -m twine upload dist/*

Shortly

Thanks for the very verbose reply, really appreciate it. Unfortunately that didn't work, as I receive the following error: ModuleNotFoundError: No module named 'src' – Thither

my guess, try to remove src in cli = src.commands.cli:cli to be cli = commands.cli:cli and then try to rebuid your package, i hope that works, let me know – Shortly

Stupid question, but how do I "run" this output? I can see that it creates a tar.gz file, but it doesn't add a binary to ./venv/bin. If I ran pip install -e ., it normally installs a binary in that directory, but I noticed it's now completely absent with this approach. – Thither

Answer 2

I had exact same problem. Your question helped me find the answer from here:

• https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

I added graft src to my manifest.in I changed (in setup.cfg under [options]) packages = find: to packages = find_namespace:

Before I added graft I was super confused. Then it could find my subpackage under my main in src

Plumb