How can I document click commands using Sphinx?

Question

Click is a popular Python library for developing CLI applications with. Sphinx is a popular library for documenting Python packages with. One problem that some have faced is integrating these two tools so that they can generate Sphinx documentation for their click-based commands.

I ran into this problem recently. I decorated some of my functions with click.command and click.group, added docstrings to them and then generated HTML documentation for them using Sphinx's autodoc extension. What I found is that it omitted all documentation and argument descriptions for these commands because they had been converted into Command objects by the time autodoc got to them.

How can I modify my code to make the documentation for my commands available to both the end user when they run --help on the CLI, and also to people browsing the Sphinx-generated documentation?

Vitriform

Answer 1

Decorating command containers

One possible solution to this problem that I've recently discovered and seems to work would be to start off defining a decorator that can be applied to classes. The idea is that the programmer would define commands as private members of a class, and the decorator creates a public function member of the class that's based on the command's callback. For example, a class Foo containing a command _bar would gain a new function bar (assuming Foo.bar does not already exist).

This operation leaves the original commands as they are, so it shouldn't break existing code. Because these commands are private, they should be omitted in generated documentation. The functions based on them, however, should show up in documentation on account of being public.

def ensure_cli_documentation(cls):
    """
    Modify a class that may contain instances of :py:class:`click.BaseCommand`
    to ensure that it can be properly documented (e.g. using tools such as Sphinx).

    This function will only process commands that have private callbacks i.e. are
    prefixed with underscores. It will associate a new function with the class based on
    this callback but without the leading underscores. This should mean that generated
    documentation ignores the command instances but includes documentation for the functions
    based on them.

    This function should be invoked on a class when it is imported in order to do its job. This
    can be done by applying it as a decorator on the class.

    :param cls: the class to operate on
    :return: `cls`, after performing relevant modifications
    """
    for attr_name, attr_value in dict(cls.__dict__).items():
        if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
            cmd = attr_value
            try:
                # noinspection PyUnresolvedReferences
                new_function = copy.deepcopy(cmd.callback)
            except AttributeError:
                continue
            else:
                new_function_name = attr_name.lstrip('_')
                assert not hasattr(cls, new_function_name)
                setattr(cls, new_function_name, new_function)

    return cls

Avoiding issues with commands in classes

The reason that this solution assumes commands are inside classes is because that's how most of my commands are defined in the project I'm currently working on - I load most of my commands as plugins contained within subclasses of yapsy.IPlugin.IPlugin. If you want to define the callbacks for commands as class instance methods, you may run into a problem where click doesn't supply the self parameter to your command callbacks when you try to run your CLI. This can be solved by currying your callbacks, like below:

class Foo:
    def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
        if isinstance(cmd, click.Group):
            commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
            cmd.commands = {}
            for subcommand in commands:
                cmd.add_command(subcommand)

        try:
            if cmd.callback:
                cmd.callback = partial(cmd.callback, self)

            if cmd.result_callback:
                cmd.result_callback = partial(cmd.result_callback, self)
        except AttributeError:
            pass

        return cmd

Example

Putting this all together:

from functools import partial

import click
from click.testing import CliRunner
from doc_inherit import class_doc_inherit


def ensure_cli_documentation(cls):
    """
    Modify a class that may contain instances of :py:class:`click.BaseCommand`
    to ensure that it can be properly documented (e.g. using tools such as Sphinx).

    This function will only process commands that have private callbacks i.e. are
    prefixed with underscores. It will associate a new function with the class based on
    this callback but without the leading underscores. This should mean that generated
    documentation ignores the command instances but includes documentation for the functions
    based on them.

    This function should be invoked on a class when it is imported in order to do its job. This
    can be done by applying it as a decorator on the class.

    :param cls: the class to operate on
    :return: `cls`, after performing relevant modifications
    """
    for attr_name, attr_value in dict(cls.__dict__).items():
        if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
            cmd = attr_value
            try:
                # noinspection PyUnresolvedReferences
                new_function = cmd.callback
            except AttributeError:
                continue
            else:
                new_function_name = attr_name.lstrip('_')
                assert not hasattr(cls, new_function_name)
                setattr(cls, new_function_name, new_function)

    return cls


@ensure_cli_documentation
@class_doc_inherit
class FooCommands(click.MultiCommand):
    """
    Provides Foo commands.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._commands = [self._curry_instance_command_callbacks(self._calc)]

    def list_commands(self, ctx):
        return [c.name for c in self._commands]

    def get_command(self, ctx, cmd_name):
        try:
            return next(c for c in self._commands if c.name == cmd_name)
        except StopIteration:
            raise click.UsageError('Undefined command: {}'.format(cmd_name))

    @click.group('calc', help='mathematical calculation commands')
    def _calc(self):
        """
        Perform mathematical calculations.
        """
        pass

    @_calc.command('add', help='adds two numbers')
    @click.argument('x', type=click.INT)
    @click.argument('y', type=click.INT)
    def _add(self, x, y):
        """
        Print the sum of x and y.

        :param x: the first operand
        :param y: the second operand
        """
        print('{} + {} = {}'.format(x, y, x + y))

    @_calc.command('subtract', help='subtracts two numbers')
    @click.argument('x', type=click.INT)
    @click.argument('y', type=click.INT)
    def _subtract(self, x, y):
        """
        Print the difference of x and y.

        :param x: the first operand
        :param y: the second operand
        """
        print('{} - {} = {}'.format(x, y, x - y))

    def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
        if isinstance(cmd, click.Group):
            commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
            cmd.commands = {}
            for subcommand in commands:
                cmd.add_command(subcommand)

        if cmd.callback:
            cmd.callback = partial(cmd.callback, self)

        return cmd


@click.command(cls=FooCommands)
def cli():
    pass


def main():
    print('Example: Adding two numbers')
    runner = CliRunner()
    result = runner.invoke(cli, 'calc add 1 2'.split())
    print(result.output)

    print('Example: Printing usage')
    result = runner.invoke(cli, 'calc add --help'.split())
    print(result.output)


if __name__ == '__main__':
    main()

Running main(), I get this output:

Example: Adding two numbers
1 + 2 = 3

Example: Printing usage
Usage: cli calc add [OPTIONS] X Y

  adds two numbers

Options:
  --help  Show this message and exit.


Process finished with exit code 0

Running this through Sphinx, I can view the documentation for this in my browser:

Vitriform

Answer 2

You can use a sphinx extension sphinx-click for this now. It can generate docs for nested commands with options and arguments description. The output will be like when you run --help.

Usage

1. Install the extension

pip install sphinx-click

2. Enable the plugin in your Sphinx conf.py file:

extensions = ['sphinx_click.ext']

3. Use plugin wherever necessary in the documentation

.. click:: module:parser
   :prog: hello-world
   :show-nested:

Example

There is simple click application, which is defined in the hello_world module:

import click


@click.group()
def greet():
    """A sample command group."""
    pass


@greet.command()
@click.argument('user', envvar='USER')
def hello(user):
    """Greet a user."""
    click.echo('Hello %s' % user)


@greet.command()
def world():
    """Greet the world."""
    click.echo('Hello world!')

For documenting all subcommands we will use code below with the :show-nested: option

.. click:: hello_world:greet
  :prog: hello-world
  :show-nested:

Before building docs make sure that your module and any additional dependencies are available in sys.path either by installing package with setuptools or by manually including it.

After building we will get this: generated docs

More detailed information on various options available is provided in documentation of the extension

Cupro