Installing data_files in setup.py with pip install -e

Question

I'm trying to provide a bash completion script for my CLI tool that is written in Python. According to the Python Packaging Authority, data_files in setup.py is exactly what I need:

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.

So I added the completion file like this:

data_files=[
    ('/usr/share/bash-completion/completions', ['completion/dotenv']),
],

and try to test it with:

pip install -e .

In my virtual environment. However, the completion script gets not installed. Did I forgot something or is pip broken? The full project can be found here

Alabama

Just a thought...did you check your output carefully? Might there be some sort of error or warning message there? – Kettle

Nothing, not even a warning or anything, even when I install with -vvv. – Alabama

To install into /usr your users must run pip install as root. I.e., you forbid to use pip install --user or installing into a virtual environment. The second problem is that bash stores completion in /usr/share/bash-completion/completions only on Linux. I.e., the package installs completion file into a wrong directory at FreeBSD or MacOS. In short, data_files is is a completely wrong approach. – Chil

See also docs.python.org/2/distutils/…: "The directory should be a relative path. It is interpreted relative to the installation prefix (Python’s sys.prefix for system installations; site.USER_BASE for user installations). Distutils allows directory to be an absolute installation path, but this is discouraged since it is incompatible with the wheel packaging format. No directory information from files is used to determine the final location of the installed file; only the name of the file is used." – Chil

Answer 1

I had the same issue and I have implemented a workaround.

It seems to me that python setup.py develop or (pip install -e .) does not run the same function than python setup.py install. In fact, I have noticed by looking in the source code that python setup.py install run build_py :

https://github.com/python/cpython/blob/master/Lib/distutils/command/build_py.py#L134 https://github.com/pypa/setuptools/blob/master/setuptools/command/build_py.py

After a few digging I have opted to override the develop command as follow. The following code is python3.6:

""" SetupTool Entry Point """
import sys
from pathlib import Path
from shutil import copy2

from setuptools import find_packages, setup
from setuptools.command.develop import develop

# create os_data_files that will be used by the default install command
os_data_files = [
    (
        f"{sys.prefix}/config",  # providing absolute path, sys.prefix will be different in venv
        [
            "src/my_package/config/properties.env",
        ],
    ),        
]


def build_package_data():
    """ implement the necessary function for develop """
    for dest_dir, filenames in os_data_files:
        for filename in filenames:
            print(
                "CUSTOM SETUP.PY (build_package_data): copy %s to %s"
                % (filename, dest_dir)
            )
            copy2(filename, dest_dir)


def make_dirstruct():
    """ Set the the logging path """
    for subdir in ["config"]:
        print("CUSTOM SETUP.PY (make_dirstruct): creating %s" % subdir)
        (Path(BASE_DIR) / subdir).mkdir(parents=True, exist_ok=True)


class CustomDevelopCommand(develop):
    """ Customized setuptools install command """

    def run(self):
        develop.run(self)
        make_dirstruct()
        build_package_data()

# provide the relevant information for stackoverflow
setup(        
    package_dir={"": "src"},
    packages=find_packages("src"),
    data_files=os_data_files,                
    cmdclass={"develop": CustomDevelopCommand},
)

Willenewillet