How to install a dependency from a submodule in Python?

Question

I have a Python project with the following structure (irrelevant source files omitted for simplicity):

myproject/
    mysubmodule/
        setup.py
    setup.py

The file myproject/setup.py uses distutils.core.setup to install the module myproject and the relevant sources. However, myproject requires mysubmodule to be installed (this is a git submodule). So what I am doing right now is:

myproject/$ cd mysubmodule
myproject/mysubmodule/$ python setup.py install
myproject/mysubmodule/$ cd ..
myproject/$ python setup.py install

This is too tedious for customers, especially if the project will be extended by further submodules in the future.

Is there a way to automate the installation of mysubmodule when calling myproject/setup.py?

Needlefish

Installing modules manually requires some skills, so, it can be too difficult for customers – Condorcet

Yes, this is exactly why I want to minimize the effort. – Needlefish

are those git submodules, or did you just use the name "submodule" for a subdirectory with a python module? – Buccal

To be honest I don't remember @Buccal – Needlefish

Answer 1

Create a package for mysubmodule with its own setup.py and let the top-level package depend on that package in its setup.py. This means you only need to make the packages / dependencies available and run python setup.py install on the top-level package.

The question then becomes how to ship the dependencies / packages to your customers but this can be solved by putting them in a directory and configuring setup.py to include that directory when searching for dependencies.

The alternative is to "vendor" mysubmodule which simply means including it all in one package (no further questions asked) and having one python setup.py install to install the main package. For example, pip vendors (includes) requests so it can use it without having to depend on that requests package.

Fluoroscope

Could you provide more details about how to specify the dependency in the setup.py of the top-level package? – Blastoderm

Saying specifically how you edit setup.py to specify that it should install the submodule as the dependency would answer the question. – Puentes

In addition to this answer, with some more details, is available here:softwareengineering.stackexchange.com/questions/365579/… – Helfant

Difficult to understand why this is the accepted answer .. the OP shows clearly that mysubmodule already has its own setup.py. What additional magic is required in the top-level setup.py? – Buccal

python setup.py is deprecated – Intoxicative

Answer 2

setuptools.find_packages() is able to discover submodules

Your setup.py should look like

from setuptools import setup, find_packages

setup(
    packages=find_packages(),
# ...
)

Doggerel