Automatically Generating Documentation for All Python Package Contents

Question

I'm trying to auto-generate basic documentation for my codebase using Sphinx. However, I'm having difficulty instructing Sphinx to recursively scan my files.

I have a Python codebase with a folder structure like:

<workspace>
└── src
    └── mypackage
        ├── __init__.py
        │   
        ├── subpackageA
        │   ├── __init__.py
        │   ├── submoduleA1
        │   └── submoduleA2
        │   
        └── subpackageB
            ├── __init__.py
            ├── submoduleB1
            └── submoduleB2

I ran sphinx-quickstart in <workspace>, so now my structure looks like:

<workspace>
├── src
│   └── mypackage
│       ├── __init__.py
│       │
│       ├── subpackageA
│       │   ├── __init__.py
│       │   ├── submoduleA1
│       │   └── submoduleA2
│       │
│       └── subpackageB
│           ├── __init__.py
│           ├── submoduleB1
│           └── ubmoduleB2
│
├── index.rst
├── _build
├── _static
└── _templates  

I've read the quickstart tutorial, and although I'm still trying to understand the docs, the way it's worded makes me concerned that Sphinx assumes I'm going to manually create documentation files for every single module/class/function in my codebase.

However, I did notice the "automodule" statement, and I enabled autodoc during quickstart, so I'm hoping most of the documentation can be automatically generated. I modified my conf.py to add my src folder to sys.path and then modified my index.rst to use automodule. So now my index.rst looks like:

Contents:

.. toctree::
   :maxdepth: 2

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

.. automodule:: alphabuyer
   :members:

I have dozens of classes and functions defined in the subpackages. Yet, when I run:

sphinx-build -b html . ./_build

it reports:

updating environment: 1 added, 0 changed, 0 removed

And this appears to have failed to import anything inside my package. Viewing the generated index.html shows nothing next to "Contents:". The Index page only shows "mypackage (module)", but clicking it shows it also has no contents.

How do you direct Sphinx to recursively parse a package and automatically generate documentation for every class/method/function it encounters, without having to manually list every class yourself?

Constrict

Answer 1

Perhaps apigen.py can help: https://github.com/nipy/nipy/tree/master/tools.

This tool is described very briefly here: http://comments.gmane.org/gmane.comp.python.sphinx.devel/2912.

Or better yet, use pdoc.

---

Update: the sphinx-apidoc utility was added in Sphinx version 1.1.

Spoken

This seems more like an afterthought for some completely unrelated project. There doesn't even seem any usage documentation for the tool itself. – Constrict

There is no way of doing what you want with vanilla Sphinx only. Something more is needed, and apigen.py is a good candidate. Why does it matter if it is "unrelated" or an "afterthought"? The tool is not neatly packaged and meticulously documented, but it's not extremely complicated either. Start by adapting the short main script, build_modref_templates.py. This script imports the ApiDocWriter class from apigen.py that does all the hard work. – Spoken

I'm concerned with it being an afterthought, because since it's an addendum to a Neuroimaging library, the developer's focus is going to be on Neuroimaging, not making apigen.py work with for the general public. However, your point about Sphinx not supporting this type of automation is well taken. I ended up going with bitbucket.org/etienned/sphinx-autopackage-script, which is dedicated to this task, although I'm sure apigen.py would probably work as well. – Constrict

Answer 2

You can try using sphinx-apidoc.

$ sphinx-apidoc --help
Usage: sphinx-apidoc [options] -o <output_path> <module_path> [exclude_paths, ...]

Look recursively in <module_path> for Python modules and packages and create
one reST file with automodule directives per package in the <output_path>.

You can mix sphinx-apidoc with sphinx-quickstart in order to create the whole doc project like this:

$ sphinx-apidoc -F -o docs project

This call will generate a full project with sphinx-quickstart and Look recursively in <module_path> (project) for Python modules.

Argentic

apidoc command does not generate index.rst file... Am I missing something? – Nomen

@Nomen index.rst and modules.rst are usually generated using sphinx-quickstart before using sphinx-apidoc. You can generate those files using only sphinx-apidoc by using the -F or -full flags. – Seal

Answer 3

Note

For Sphinx (actually, the Python interpreter that executes Sphinx) to find your module, it must be importable. That means that the module or the package must be in one of the directories on sys.path – adapt your sys.path in the configuration file accordingly

So, go to your conf.py and add

import an_example_pypi_project.useful_1
import an_example_pypi_project.useful_2

Now your index.rst looks like:

.. toctree::
   :glob:

   example
   an_example_pypi_project/*

and

make html

Endopeptidase

Answer 4

From Sphinx version 3.1 (June 2020), if you're happy to use sphinx.ext.autosummary to display summary tables, you can use the new :recursive: option to automatically detect every module in your package, however deeply nested, and automatically generate documentation for every attribute, class, function and exception in that module.

See my answer here: https://mcmap.net/q/117428/-automatically-document-all-modules-recursively-with-sphinx-autodoc

Acidulant