Sphinx cannot find my python files. Says 'no module named ...'

Question

I have a question regarding the Sphinx autodoc generation. I feel that what I am trying to do should be very simple, but for some reason, it won't work.

I have a Python project of which the directory is named slotting_tool. This directory is located at C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool

I set up Sphinx using sphinx-quickstart. Then my directory structure (simplified) is as follows:

slotting_tool/
|_ build/
|_ source/
|___ conf.py
|___ index.rst
|_ main/
|___ run_me.py

Now, I set the root directory of my project to slotting_tool by adding the following to the conf.py file.

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

Next, I update my index.rst file to look like this:

.. toctree::
   :maxdepth: 2
   :caption: Contents:

.. automodule:: main.run_me
   :members:

When trying to build my html using the sphinx-build -b html source .\build command, I get the following output, with the no module named error:

(base) C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool>sphinx-build -b html source .\build
Running Sphinx v1.8.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [] 0 added, 1 changed, 0 removed
reading sources... [100%] index
WARNING: autodoc: failed to import module 'run_me' from module 'main'; the following exception was raised:
No module named 'standalone'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in build.

There are no HTML pages that refer to run_me.py in build. I have tried setting my root directory to all different kinds of directories and I have tried replacing all dots . with backslashes \ and so forth, but can't seem to find out what I'm doing wrong.

By the way, the statement that standalone is not a module is in fact true, it is just a directory without an __init__.py. Don't know if that might have caused some trouble?

Anyone have an idea?

Handmaid

I think your intuition is correct. Add an empty init.py and see if the import warning goes away. – Nitaniter

I tried it with the __init__.py added, but this does not resolve the error either. Do you know of anything else that may cause this error? – Handmaid

Without seeing your code, I assume that run_me.py tries to import standalone but fails. I'd need to see the code. – Nitaniter

Sphinx will attempt to import all your code, just like running it in a Python interpreter. Fix your imports or specify the path to the non-importable code with sys in conf.py. – Nitaniter

I finally fixed the no module named error. Indeed, the imports require that you define the root directory in conf.py to such a low level, that it is at least below the - in my case - standalone folder. I thus set my root directory to picnic-data-shared-tools and each time I call to a .py file, I move deeper into the project structure to access it. However, generating the html build files, I do not get an error this time, but just an empty html. Will try to find out what is happening here, otherwise, I'll be back. – Handmaid

Your module must be in a Python package that Sphinx can import. You need to put the module in a folder along with an empty __init__.py file (thus making it a Python package), and configure that path in your conf.py so Sphinx can import it. – Nitaniter

Answer 1

This is the usual "canonical approach" to "getting started" applied to the case when your source code resides in a src directory like Project/src instead of simply being inside the Project base directory.

Follows these steps:

1. Create a docs directory in your Project directory (it's from this docs directory the commands in the following steps are executed).

2. sphinx-quickstart (choose separate source from build. Places .html and .rst files in different folders).

3. sphinx-apidoc -o ./source ../src

4. make html

This would yield the following structure (provided you .py source files reside in Project/src):

Project
|
├───docs
│   │   make.bat
│   │   Makefile
│   │
│   ├───build
│   └───source
│       │   conf.py
│       │   index.rst
│       │   modules.rst
│       │   stack.rst
│       │
│       ├───_static
│       └───_templates
└───src
        stack.py

In your conf.py you'd add (after step 2):

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))

Also include in conf.py:

extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']

And in index.rst you'd link modules.rst:

Welcome to Project's documentation!
================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules
      
   
Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

Your stack.rst and modules.rst were auto-generated by sphinx-apidoc, no need to change them (at this point). But just so you know this is what they look like:

stack.rst:

stack module
============

.. automodule:: stack
   :members:
   :undoc-members:
   :show-inheritance:

modules.rst:

src
===

.. toctree::
   :maxdepth: 4

   stack

---

---

After `make html` open `Project/docs/build/index.html` in your browser, the results:

and:

Euboea

At conf.py the command that works for me is sys.path.insert(0, os.path.abspath(os.path.join('..', '..'))) otherwise I am getting the error WARNING: autodoc: failed to import module 'source'; the following exception was raised: No module named 'source'. also you are missing __init__.py file at ./src/ – Ithyphallic

@YogevNeumann then you made a mistake because the example is correct. __init__.py was not included because src itself is meant to be a base directory, that's the recommended practice. – Euboea

@YogevNeumann the mistake you are doing is putting an __init__.py inside src, that's not how a source layout is supposed to be used. You can put modules or packages inside src but it's not meant to be a package itself. That's the established practice that everyone uses, see the docs. – Euboea

@Euboea If i do that i get the error message No module named 'src' – Lurleen

@Lurleen the answer works exactly as it's shown, when something doesn't work it's because you changed something relative to the answer. Do you have a src directory? What's the complete error message? – Euboea

@Euboea thanks for the quick replay, I made it work by not only including 'src' but also the parent folder like yogev said, so including both. – Lurleen

@Lurleen if you've missed it I added a link to the docs about the src layout in the comments. If you're using a src layout it's expected you know what it means in the context of the question, here's a simpler example using a flat layout. – Euboea

this answer deserves a nobel prize – Akkadian

The worst part of python is the tooling ecosystem. – Psychosurgery

Answer 2

Let's take an example with a project: dl4sci-school-2020 on master branch, commit: 6cbcc2c72d5dc74d2defa56bf63706fd628d9892:

├── dl4sci-school-2020
│   ├── LICENSE
│   ├── README.md
│   ├── src
│   │   └── __init__.py
│   └── utility
│       ├── __init__.py
│       └── utils.py

and utility package has a utils.py module:

Follow this process(FYI, I'm using sphinx-build 3.1.2):

1. create a docs/ directory under you project:

mkdir docs
cd docs

2. start sphinx within docs/, and just pass your project_name, your_name & version of your choice and rest keep defaults.

sphinx-quickstart

you will get below auto-generated in your docs/ folder

├── docs
│   ├── Makefile
│   ├── build
│   ├── make.bat
│   └── source
│       ├── _static
│       ├── _templates
│       ├── conf.py
│       └── index.rst

Since, we created a separate docs directory so we need sphinx find where to find build files and python src module. So, edit the conf.py file, you can use my conf.py file too

import os
import sys
basedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
sys.path.insert(0, basedir)

Now, to enable access to nested multiple packages & modules if any, you need to edit index.rst file.

.. toctree::
   :maxdepth: 2
   :caption: Description of my CodeBase:

   modules

The modules picks up content from modules.rst file which we will create below: Make sure you're still in doc/ to run the below command

sphinx-apidoc -o ./source ..

The output you get:

├── docs
│   ├── Makefile
│   ├── build
│   ├── make.bat
│   └── source
│       ├── _static
│       ├── _templates
│       ├── conf.py
│       ├── index.rst
│       ├── modules.rst
│       ├── src.rst
│       └── utility.rst

now run:

make html

Now, go and open in browser of your choice,

file:///<absolute_path_to_your_project>/dl4sci-school-2020/docs/build/html/index.html

have you beautiful documentation ready

https://i.sstatic.net/5pvLu.jpg

FYI, You can switch any theme of your choice, I found sphinx_rtd_theme and extension sphinxcontrib.napoleon super dope!. Thanks to their creators, so I used it.

below does the work!

pip install sphinxcontrib-napoleon
pip install sphinx-rtd-theme

You can host your documentation it on readthedocs enjoy documenting your code!

Left

Warning to readers: This answer is wrong and you'll me mislead if you follow it. The answer fails to use a src layout or a flat layout, see the docs src layout vs flat layout. If you want to use a src layout follow the top voted answer, if you want to use a flat layout see this answer. – Euboea

Answer 3

sys.path.insert(0, os.path.abspath('../..'))

That's not correct. Steve Piercy's comment is not entirely on point (you don't need to add a __init__.py since you're using a simple module) but they're right that autodoc will try to import the module and then inspect the content.

However assuming your tree is

doc/conf.py
src/stack.py

then you're just adding the folder which contains your repository to the sys.path which is completely useless. What you need to do is add the src folder to sys.path, such that when sphinx tries to import stack it finds your module. So your line should be:

sys.path.insert(0, os.path.abspath('../src')

(the path should be relative to conf.py).

Of note: since you have something which is completely synthetic and should contain no secrets, an accessible repository or a zip file of the entire thing makes it much easier to diagnose issues and provide relevant help: the less has to be inferred, the less can be wrong in the answer.

Carcinogen

Answer 4

IMHO running pip install --no-deps -e . in the top project folder (or where ever setup.py is) to get an "editable" install is a better alternative to get your package modules on the PYTHONPATH than altering it in docs/conf.py using sys.path.

Martguerita

I have reached the same conclusion -- in fact, I would omit the --no-deps, since Sphinx directives like automodule will actually import the module using standard Python facilities, and if some dependencies are missing, you will get ImportErrors. – Watusi