Is it possible to include subdirectories using dist utils (setup.py) as part of package data?

Question

Basically my python package is setup like:

module
\_examples
  \_folder1
     \_file1.py
     \_file2.py
  \_folder2
    \_file1.py
    \_file2.py

Basically I want to just use:

package_data  = { 
            'module': ['examples/*'],
  },

because my project always has people adding examples and I want it to be easy to list them from within my application. I can get it work for any FILE within examples, but not re-curse down through sub-directories. Is this possible?

Precast

Answer 1

I believe what you're looking for is something like this for you setup.py, which will recursively find any packages in the project, also be sure and include __init__.py files to subdirectories for each package you want.

from setuptools import setup, find_packages

setup(name='MySoftware',
      packages=find_packages()
)

Weinreb

Be careful here, because you might accidentally include test directories, etc. unless you explicitly exclude=... these. – Alexanderalexandr

worked in combination with MANIFEST.in for non-python files for me – Bish

Answer 2

In my package root folder, I have a setup.py file see doc
In this file, I have the following code:

from setuptools import setup

with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='package name',
    version='0.4.1',
    description='short description',
    long_description=long_description,
    long_description_content_type="text/markdown",
    url='repository url',
    author='My name',
    author_email='[email protected]',
    license='MIT',
    packages=['PackageName','PackageName.SubModule'],
    zip_safe=False,
    install_requires=[
        'dependecy1',
    ],
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3.7'
    ]
)

The interesting part to answer the question, here is : packages=['PackageName','PackageName.SubModule'],

By following this syntax, you can include sub-modules to your main package distribution.

More info about all the others arguments can be found in the doc.

Rent

Out of all the answers here, I think this is the cleanest way – Encampment

Answer 3

You'll have to use a MANIFEST.in file for that.

I believe you'll want something like this:

$ cat MANIFEST.in
recursive-include examples/ *.py

Epizoon

I think you can also directly use Python functions in setup.py to find files and directories, but I can't quite find the exact answer at the moment. – Spaceband

Hrm, I will have to give it a try as I did: recursive-include examples *.py and didn't use the /. I read that MANIFEST.in is being deprecated though. I think I may just have to follow the python function call method. – Precast

Oh… Well, if you figure out how to do it with Python, please post here. I'd like to know. – Epizoon

where is the location of the Manifest.in file? Thank you – Rondelle

Answer 4

Yes, you can include all the subdirectories.

You just need to pass the below args to setup() function:

packages=find_packages()

include_package_data=True

Along with this you need to have a MANIFEST.in file, with contents as

recursive-include examples *

This ensures all the files are recursively included.

If you want to exclude certain extensions specifically, you can do so by specifying exclude array in the find_packages() argument.

Ex: To exclude .txt files

packages=find_packages(exclude=['.txt'])

You can read more about include_package_data here.

And also here is another link which tells you when you should not use include_package_data

Shoemake

Answer 5

None of the suggested answers worked for me in a similar situation.

I needed to make a distribution with my package, which included several sub-modules in a sub-directory, so that these were the files I needed to go into sdist:

ipyexperiments/*py
ipyexperiments/utils/*py

and no matter what I tried, the subdir utils's modules were not getting included in sdist.

What worked for me is leaving config.py's default:

# config.py
from setuptools import setup, find_packages
[...]
setup(
    packages = find_packages(),
    [...]
)

but adding to MANIFEST.in:

# MANIFEST.in
graft ipyexperiments

and everything under ipyexperiments was included.

If you don't already have MANIFEST.in, create it at the same directory as config.py.

I also added to MANIFEST.in

prune tests
global-exclude *.py[co]

to exclude all of tests directory and any unwanted *pyc and *.pyo files anywhere.

Appledorf

where does your MANIFEST.in file live? At the same level as setup.py or at the top of the package you're making? – Transmogrify

that's correct. I edited the reply to reflect that. thank you for asking. – Appledorf

Answer 6

You can make use of include parameter of find_packages():

...
setup(name="module",
      packages=find_packages(include=('module*',)),
...

Gardal

In my case I had to use find_packages(include=('module/*',)) – Eyepiece

Answer 7

Following what David Wolever said, just to make it a little more clear. If you want to include everything under a sub-directory folder you have to explicitly specify each file in the MANIFEST.in,

recursive-include examples/ *.py *.png *.sh etc.....

It would be nice if the manifest.in would just understand examples/ and include everything but oh well.

Capita

It does support including all files below a certain path now with graft some-dir/. – Miasma

Answer 8

All other answers seemed to complicated. This worked with version 9.3.15:

package_data  = { 
            'module': ['examples/**'],
  },

Note the ** instead of *. No __init__.py was required in any folders.

Patina

Answer 9

The simple answer that worked for me is to just include all the directories in the packages= option in the setup call:

 packages=[“examples”, “examples/_folder1”, “examples/_folder2”]

Fritter