Install by default, "optional" dependencies in Python (setuptools)

Question

Is there a way to specify optional dependencies for a Python package that should be installed by default from pip but for which an install should not be considered a failure if they cannot be installed?

I know that I can specify install_requires so that the packages will be installed for the 90% of users using OSes that can easily install certain optional dependencies, and I also know I can specify extra_require to specify that users can declare they want a full install to get these features, but I haven't found a way to make a default pip install try to install the packages but not complain if they cannot be installed.

(The particular package I'd like to update the setuptools and setup.py for is called music21 for which 95% of the tools can be run without matplotlib, IPython, scipy, pygame, some obscure audio tools etc. but the package gains extra abilities and speed if these packages are installed, and I'd prefer to let people have these abilities by default but not report errors if they cannot be installed)

Collection

And of course, nowadays, solutions that work with pyproject.toml would be welcomed as well, as I've dropped setuptools and setup.py from my workflow. – Collection

Answer 1

Not a perfect solution by any means, but you could setup a post-install script to try to install the packages, something like this:

from distutils.core import setup
from distutils import debug


from setuptools.command.install import install
class PostInstallExtrasInstaller(install):
    extras_install_by_default = ['matplotlib', 'nothing']

    @classmethod
    def pip_main(cls, *args, **kwargs):
        def pip_main(*args, **kwargs):
            raise Exception('No pip module found')
        try:
            from pip import main as pip_main
        except ImportError:
            from pip._internal import main as pip_main

        ret = pip_main(*args, **kwargs)
        if ret:
            raise Exception(f'Exitcode {ret}')
        return ret

    def run(self):
        for extra in self.extras_install_by_default:
            try:
                self.pip_main(['install', extra])
            except Exception as E:
                print(f'Optional package {extra} not installed: {E}')
            else:
                print(f"Optional package {extra} installed")
        return install.run(self)


setup(
    name='python-package-ignore-extra-dep-failures',
    version='0.1dev',
    packages=['somewhat',],
    license='Creative Commons Attribution-Noncommercial-Share Alike license',
    install_requires=['requests',],
    extras_require={
        'extras': PostInstallExtrasInstaller.extras_install_by_default,
    },
    cmdclass={
        'install': PostInstallExtrasInstaller,
    },
)

Commit

This seems like a great idea I hadn’t thought of. Worthy of the bounty if nothing simpler emerges. – Collection

Beware that post-install commands are useless with pip, except that you explicilty instruct your users to install via pip install pkgname --no-binary=pkgname. This will only work when invoking python setup.py install directly. – Ferguson

Please don't recommend that users import pip: it is a command line tool, and doesn't support any importable API. Doing so causes breakage and confusion for users, and headache for pip developers when they need to move things around internally (hence your try/except -- the pip._internal module indicates that this is internal to pip and should not be imported.) – Microscope

Answer 2

The simplest way to do this is by adding a custom install command that simply shells out to pip to install the "optional" packages. In your setup.py:

import sys
import subprocess
from setuptools import setup
from setuptools.command.install import install

class MyInstall(install):
    def run(self):
        subprocess.call([sys.executable, "-m", "pip", "install", "whatever"])
        install.run(self)

setup(
    ...

    cmdclass={
        'install': MyInstall,
    },
)

Like hoefling said above, this will only work if you publish a source distribution (.tar.gz or .zip). It will not work if you publish your package as a built distribution (.whl).

Microscope

Alas, I no longer use setup.py, having recently moved to pyproject.toml and hatchling – Collection

Answer 3

This might be what you are looking for. It's appears to be a built in feature of setup tools that allows you to declare "optional dependencies".

https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies

Ex

setup(
    name="Project-A",
    ...
    extras_require={
        'PDF':  ["ReportLab>=1.2", "RXP"],
        'reST': ["docutils>=0.3"],
    }
)

Poussette

these unfortunately do not install by default. They're a great way for power users to decide what to install, but not helpful for most users. – Collection