Importing custom module into jupyter notebook

Question

Yes, I know this is a recurrent question but I still couldn't find a convincing answer. I even read at https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html but could not find out how to solve the problem:

I'm running python 3.6 project that includes jupyter (ipython) notebooks. I want the notebook to import a custom local helpers.py package that I will probably use also later in other sources.

The project structure is similar to:

my_project/
│
├── my_project/
│   ├── notebooks/
│       └── a_notebook.ipynb
│   ├── __init__.py     # suppose to make package `my_project` importable
│   └── helpers.py
│
├── tests/
│   └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py

When importing helpers in the notebook I get the error:

----> 4 import helpers

ModuleNotFoundError: No module named 'helpers'

I also tried from my_project import helpers and I get the same error ModuleNotFoundError: No module named 'my_project'

I finally (and temporarily) used the usual trick:

import sys
sys.path.append('..')
import helpers

But it looks awful and I'm still looking for a better solution

Prothalamion

Have you tried form .. import helpers or something like that? Try relative imports – Stier

Have you tried moving helpers.py to the notebooks directory? – Bolster

Of course I could do that @NielsHenkens and it'll work but I don't like this solution since I want to reuse the functions in helpers.py in other part of my_project, not only in notebooks that are only exploratory phases. helpers are not specific to their use in the notebooks. – Prothalamion

@RaydelMiranda: relative import like the one you suggest returns the error ValueError: attempted relative import beyond top-level package. I've also tried from ..my_project import helpers – Prothalamion

Answer 1

One can tell python where to look for modules via sys.path. I have a project structure like this:

project/
    │
    ├── src/
    │    └── my_module/
    │        ├── __init__.py       
    │        └── helpers.py
    ├── notebooks/
    │   └── a_notebook.ipynb
    ...

I was able to load the module like so:

import sys
sys.path.append('../src/')

from my_module import helpers

One should be able load the module from wherever they have it.

Peekaboo

Answer 2

Here I could find several solutions. Some of them are similar to the ones answered before: https://mg.readthedocs.io/importing-local-python-modules-from-jupyter-notebooks/index.html

Prothalamion

It would be better to post one or more of the solutions from the link directly here as an answer. See Why is linking bad? or Are answers that just contain links elsewhere really “good answers”? – Axillary

Answer 3

If you move the notebooks directory out one level, and then explicitly import your module from the package, that should do it. So your directory would look like this:

my_project/
│
├── my_project/
│   ├── __init__.py       
│   └── helpers.py
├── notebooks/
│   └── a_notebook.ipynb
...

and then your import statement within the notebook would be:

from my_project import helpers.

Halliburton

Have you run the setup.py file such that the my_project package has been installed? – Halliburton

No, I don't use setup.py. Should I? – Prothalamion

Well, the only reason for you to have a setup.py file and all the __init__.py files in this folder is if you want to use my_project like a package. In order to install that package into Python, you would need to run python setup.py. However, if you don't want to use my_project like a package, but rather just as a collection of modules, then there is no need to have setup.py or __init__.py in this folder. In this case, you would move a_notebook.ipynb into the my_project folder (next to helpers.py) and just do import helpers within the notebook. – Halliburton

@Halliburton unfortunately in jupyter notebook you cant import the module as easy just like that even if you put your own module in the same file. the output always showed 'no input name 'blabla' '. this is really hard if you want to do unittest. otherwise anyone can suggest other IDE that is easier to import own module? – Edmonton

Tried exactly this example and it did not work for me. – Gomel

Answer 4

Try the following line:

from my_project.helpers import what_you_need

This line should also work:

import my_project.helpers

Trimming

Answer 5

This worked for me.

import sys

MODULE_FULL_PATH = '/<home>/<username>/my_project/my_project'

sys.path.insert(1, MODULE_FULL_PATH)

from my_module import helpers

Derron

Answer 6

I think you need a __init__.py module in the notebooks/ directory. I haven't really used Jupyter notebooks before so I could be wrong. You may also need to try changing your import statement to:

import .. helpers

to indicate that the import statement is for a local package that is located in the parent directory of the Jupyter notebook.

Yolanda

for first suggestion: __init__.py enables the directory that contains it to be a package that can be imported. The purpose here is not to make notebooks a package to import. About relative imports you suggest, the correct syntax is from .. import helpers. It gives the error (I've also tried from ..my_project import helpers): ValueError: attempted relative import beyond top-level package – Prothalamion

This is incorrect, in any case, it should be import ..helpers for one level up – Billyebilobate

This leads to: SyntaxError: invalid syntax – Emeliaemelin

Answer 7

If you are on a Unix/Linux system another elegant solution may be creating a "soft link" to the module file helpers.py that you would like to use. Change to the notebooks directory and create the link to the module file this way:

cd notebooks; ln -fs ../my_project/helpers.py .

This "soft link" is essentially a pointer (a shortcut) to the original target file. With the link in place you will be import your module file as usual:

import helpers

Nonunionism

Answer 8

It is probably too late for answer, but the solution for me was pretty much simple. My problem was that I first created a project in Jupiter, and then add e helpers.py with my finctions. So basically your Jupiter notebook won't see it untill you restart the Kernel in your Jupiter Notebook. Make sure you did that, and once you do, your imports strat works no problems.

Refractor