How to properly deal with optional features in python

Question

I'm working on python packages that implement scientific models and I'm wondering what is the best way to handle optional features. Here's the behavior I'd like: If some optional dependencies can't be imported (plotting module on a headless machine for example), I'd like to disable the functions using these modules in my classes, warn the user if he tries to use them and all that without breaking the execution. so the following script would work in any cases:

mymodel.dostuff()
mymodel.plot() <= only plots if possible, else display log an error 
mymodel.domorestuff() <= get executed regardless of the result of the previous statement

So far the options I see are the following:

• check in the __init __.py for available modules and keep a list of them (but how to properly use it in the rest of the package?)
• for each function relying on optional dependencies have a try import ... except ... statement
• putting functions depending on a particular module in a separated file

These options should work, but they all seem to be rather hacky and hard to maintain. what if we want to drop a dependency completely? or make it mandatory?

Hohenlinden

Answer 1

The easiest solution, of course, is to simply import the optional dependencies in the body of the function that requires them. But the always-right PEP 8 says:

Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.

Not wanting to go against the best wishes of the python masters, I take the following approach, which has several benefits...

First, import with an try-except

Say one of my functions foo needs numpy, and I want to make it an optional dependency. At the top of the module, I put:

try:
    import numpy as _numpy
except ImportError:
    _has_numpy = False
else:
    _has_numpy = True

Here (in the except block) would be the place to print a warning, preferably using the warnings module.

Then throw the exception in the function

What if the user calls foo and doesn't have numpy? I throw the exception there and document this behaviour.

def foo(x):
    """Requires numpy."""
    if not _has_numpy:
        raise ImportError("numpy is required to do this.")
    ...

Alternatively you can use a decorator and apply it to any function requiring that dependency:

@requires_numpy
def foo(x):
    ...

This has the benefit of preventing code duplication.

And add it as an optional dependency to your install script

If you're distributing code, look up how to add the extra dependency to the setup configuration. For example, with setuptools, I can write:

install_requires = ["networkx"],

extras_require = {
    "numpy": ["numpy"],
    "sklearn": ["scikit-learn"]}

This specifies that networkx is absolutely required at install time, but that the extra functionality of my module requires numpy and sklearn, which are optional.

---

Using this approach, here are the answers to your specific questions:

• What if we want to make a dependency mandatory?

We can simply add our optional dependency to our setup tool's list of required dependencies. In the example above, we move numpy to install_requires. All of the code checking for the existence of numpy can then be removed, but leaving it in won't cause your program to break.

• What if we want to drop a dependency completely?

Simply remove the check for the dependency in any function that previously required it. If you implemented the dependency check with a decorator, you could just change it so that it simply passes the original function through unchanged.

This approach has the benefit of placing all of the imports at the top of the module so that I can see at a glance what is required and what is optional.

Tactless

Thank you for you answer! Then, if I don't want to break the execution, I'd use a warning instead of raising ImportError. Is that acceptable? – Hohenlinden

'try-import' is 100% the right way to do optional dependencies in Python. I've used it myself many times, and it's very commonly seen in the wild too. It's way more readable than hiding imports inside a function. +1 – Huneycutt

The onus is usually on the user to avoid breaking execution, except in rare cases. If foo absolutely requires numpy to run, then you should say as much in the documentation: that an ImportError will be raised if this is not the case. Then the user can catch this exception in her code if she so pleases. One case where you might prefer a warning: where numpy might provide an alternative, faster implementation of the algorithm. If the user has numpy, you might use it, otherwise you'd print a warning, fall back to builtin functionality, and continue on without breaking execution. – Tactless

Answer 2

I would use the mixin style of composing a class. Keep optional behaviour in separate classes and subclass those classes in your main class. If you detect that the optional behaviour is not possible then create a dummy mixin class instead. For example:

model.py

import numpy
import plotting

class Model(PrimaryBaseclass, plotting.Plotter):
    def do_something(self):
        ...

plotting.py

from your_util_module import headless as _headless
__all__ = ["Plotter"]
if _headless:
    import warnings
    class Plotter:
        def plot(self):
            warnings.warn("Attempted to plot in a headless environment")
else:
    class Plotter:
        """Expects an attribute called `data' when plotting."""
        def plot(self):
            ...

Or, as an alternative, use decorators to describe when a function might be unavailable.

eg.

class unavailable:

    def __init__(self, *, when):
        self.when = when

    def __call__(self, func):
       if self.when:
           def dummy(self, *args, **kwargs):
               warnings.warn("{} unavailable with current setup"
                   .format(func.__qualname__))
           return dummy
       else:
           return func

class Model:
    @unavailable(when=headless)
    def plot(self):
        ...

Transfinite

Answer 3

Based on jme's answer, I ended up writing a parameterized decorator:

from functools import wraps
from importlib.util import find_spec


def depends_on_optional(module_name: str):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            spec = find_spec(module_name)
            if spec is None:
                raise ImportError(
                    f"Optional dependency {module_name} not found ({func.__name__})."
                )
            else:
                return func(*args, **kwargs)

        return wrapper

    return decorator

I am using it to get an ImportError when plot_with_matplotlib is called at runtime despite the warning that matplotlib is not found:

import warnings

try:
    import matplotlib.pyplot as plt
except ImportError:
    warnings.warn("matplotlib not found, plotting functions will not work")

from my_decorator import depends_on_optional


@depends_on_optional("matplotlib.pyplot")
def plot_with_matplotlib():
    pass

To get the exact behavior the OP wants, you can move the warning into the decorator in place of the ImportError.

I also adapted the requirements.txt based on this answer with an optional dependency:

matplotlib[matplotlib]

With this setup I don't have to keep track of any import state (like _has_matplotlib) myself!

Scaremonger