sys.path different in Jupyter and Python - how to import own modules in Jupyter?

Question

In Jupyter my own little module is not loaded but in python/bpython is everything is fine. When typing

import sys
print(sys.path)

the path to my module will not in show in Jupyter but in python/bpython it is still there.

I am using:

1. PYTHONPATH in .bashrc to include my module,
2. Jupyter and bpython inside a virtualenv.

The most similar questions is this Cannot import modules in jupyter notebook; wrong sys.path

How to configure Jupyter to load my modules automagically?

Japonica

have you tried running sys.path.append('/path/to/your/code') within the Jupyter ipython shell or notebook? If that works, you can add this command to the ipython profile. – Textile

thank you for your hint with the ipython profile. I was not aware that ipython/jupyter does it own thing. A good instruction what to do is found here lucypark.kr/blog/2013/02/10/… – Japonica

One general comment is to run both through an anaconda environment. That way you a) do not mess with system files, b) have both running the same versions of packages and avoid confusions. Also for your module, it's quite easy to write a setup.py and install an ediatble version of it in the environment. That way you can do ``` import my module``` straight away. – Rivalee

Answer 1

Here is what I do on my projects in jupyter notebook,

import sys
sys.path.append("../") # go to parent dir
from customFunctions import *

Then, to affect changes in customFunctions.py,

%load_ext autoreload
%autoreload 2

Estes

Do you really want to use append? I would suggest that it is better to prepend; e.g. sys.path.insert(0, ".."). If you were to prepend ".." to your path, then the parent directory would be searched first, which is arguably what you should want to happen. Also, this is consistent with how Jupyter notebook modifies the path -- it puts the path of the current notebook directory as the first item on the path. – Kazak

What does "affect changes in customFunctions.py" mean? Do you mean you modified customFunctions.py after starting the notebook? – Diversion

@Diversion it means; when/if you make changes on that file you don't need to re-start the notebook. The notebook reloads automatically the objects after every changes. – Estes

Answer 2

Jupyter is base on ipython, a permanent solution could be changing the ipython config options.

Create a config file

$ ipython profile create
$ ipython locate
/Users/username/.ipython

Edit the config file

$ cd /Users/username/.ipython
$ vi profile_default/ipython_config.py

The following lines allow you to add your module path to sys.path

c.InteractiveShellApp.exec_lines = [
    'import sys; sys.path.append("/path/to/your/module")'
]

At the jupyter startup the previous line will be executed

Here you can find more details about ipython config https://www.lucypark.kr/blog/2013/02/10/when-python-imports-and-ipython-does-not/

Cruller

While this works, it looks like it does nothing but executing the append for users at startup. Is there a better way where we can directly modify the path displayed by sys.path? – Doublecross

For me it worked after using absolute paths (so '/home/user/module' instad of '~/module' ) – Charqui

Answer 3

Jupyter has its own PATH variable, JUPYTER_PATH.

Adding this line to the .bashrc file worked for me:

export JUPYTER_PATH=<directory_for_your_module>:$JUPYTER_PATH

Haplite

I have tried this several times in the past and can't remember it ever working. This is definitely not the correct solution. – Etude

This doesn't work; it doesn't allow one to import python modules from a path added to JUPYTER_PATH. Looking at the docs it seems to be for Jupyter "data", such as "kernelspecs, nbextensions, or voila templates". – Arathorn

Answer 4

Suppose your project has the following structure and you want to do imports in the notebook.ipynb:

/app
  /mypackage
    mymodule.py
  /notebooks
    notebook.ipynb

If you are running Jupyter inside a docker container without any virtualenv it might be useful to create Jupyter (ipython) config in your project folder:

/app
  /profile_default
    ipython_config.py

Content of ipython_config.py:

c.InteractiveShellApp.exec_lines = [
    'import sys; sys.path.append("/app")'
]

Open the notebook and check it out:

print(sys.path)

['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages/IPython/extensions', '/root/.ipython', '/app']

Now you can do imports in your notebook without any sys.path appending in the cells:

from mypackage.mymodule import myfunc

Mulch

Answer 5

The verified solution doesn't work for me, since my notebook is not in my sys.path. This works however;

import os,sys
sys.path.insert(1, os.path.join(os.getcwd()  , '..'))

Johnnajohnnie

Answer 6

I don't like sys.path.append("../"). This actually adds string ../ to the path and not the absolute path.

• This causes path differences while loading data files (like csv). If we go down that route, we need to add relative paths to those file paths as well.
• This causes __file__ reference to have relative .. in them. If there are logics written around __file__ reference those may break.

The following approach adds absolute path to the sys.path and I didn't face any issue:

import sys
import os

sys.path.insert(0, os.path.abspath('..'))

Now everything works same way as it works from normal python & jupyter notebook.

We can check the sys path and validate with following :

# Testing
import sys
for p in sys.path:
    print(p)

We will still have issues with accessing file and their paths used within our modules. This is because file io like open uses current directory based paths.

Best solution would be to clean up the way we construct file paths to have relative path from that particular python module.

For example:

def get_transactions():    
    transaction_path = Path(__file__).parent.parent.parent / 'data_source/some_input.csv'
    print(transaction_path) 
    transactions_df = pd.read_parquet(path=transaction_path) if os.path.isfile(transaction_path) else None
    return transactions_df

Aurie

Answer 7

You can use absolute imports:

/root
    /app
      /config
        newfile.py
      /source
        file.ipynb

# In the file.ipynb importing the newfile.py file
import os
os.chdir(ROOT_PATH)
from app.config import newfile

Indictable

This does not work as the notebook thinks the root folder is the folder its in. It can't see root or app or config or source – Jongjongleur

You're right. I had to do os.chdir(ROOT_PATH) inside the notebook then I could import. Updated my answer and just tested inside a notebook. – Indictable