Import py file in another directory in Jupyter notebook

Question

My question is related to this. I am using Python 3.6 in Jupyter Notebook. My project directory is /user/project. In this directory I'm building a number of models and each has its own folder. However, there is a common functions.py file with functions that I want to use across all models. So I want to keep the functions.py file in /user/project but be able to call it from an .ipynb file in /user/project/model1, /user/project/model2, etc... How can I do this?

Birthmark

Did the solution in the question you linked not work for you? What error did you receive? Is this of any help: https://mcmap.net/q/202677/-how-to-execute-a-py-file-from-a-ipynb-file-on-the-jupyter-notebook/8146556 – Jacquelinjacqueline

Python module/package/imports management is such a flexible clusterfuck, so these questions are always helpful :-) – Austroasiatic

Answer 1

There is no simple way to import python files in another directory. This is unrelated to the jupyter notebook

Here are 3 solutions to your problem

1. You can add the directory containing the file you want to import to your path and then import the file like this:

import sys  
sys.path.insert(1, '/path/to/application/app/folder')

import file

2. You can create a local module by having an empty __init__.py file in the folder you want to import. There are some weird rules regarding the folder hierarchy that you have to take into consideration.

3. You can create a module for the file you wish to import and install it globally.

Eschew

I believe the "weird rules" link you posted has moved here docs.python.org/3/tutorial/modules.html – Defecate

Please could you or someone else expand on the third solution here: "create a module for the file you wish to import and install it globally". Thanks. – Tannie

Option 1 might be better as sys.path.insert(1, '/path/to/application/app/folder') because path[0] is reserved for the script path (or '' in REPL). – Allinclusive

Answer 2

Assuming you have a folder name Jupyter and you wish to import modules (employee) from another folder named nn_webserver.

visualizing it:

do this:

import sys
import os

module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path+"\\nn_webserver")

from employee import motivation_to_work

see additional information here from @metakermit

Intermolecular

I prefer this relative path to the other answer, which uses an absolute path. The relative path will remain valid even if you install this entire project on a different system, whereas an absolute path would need to be adjusted. – Scarab

Note that this only works on Windows due to not using os.path.sep – Behn

Answer 3

I've been thinking about this problem because I don't like the sys.path.append() answers. A solution I propose uses the built-in Jupyter magic command to change the current working directory. Assuming you have this file structure:

project
├── model1
|   └── notebook1.ipynb
├── model2
|   └── notebook2.ipynb
└── functions.py

Whether you wanted to import functions from notebook1.ipynb or notebook2.ipynb, you could simply add a cell with the following line before the cell that has your package imports:

%cd ..

This changes the current working directory to the parent directory of the notebook, which then adds the path of the functions module to the default locations that Python will check for packages. To import functions:

import functions

This would work similarly if you had multiple modules in the same package directory that you wanted to import:

project
├── model1
|   └── notebook1.ipynb
├── model2
|   └── notebook2.ipynb
└── package
    ├── functions1.py
    └── functions2.py 

You can import both modules functions1 and functions2 from package like this:

from package import functions1, functions2

EDIT: As pointed out below, the local imports will no longer work if the cell containing the magic command is run more than once (the current working directory will be changed to the directory above at each rerun of the command). To prevent this from happening, the %cd .. command should be in its own cell (not in the same cell as the imports) at the top of the notebook and before the imports so it won't be run multiple times. Restarting the kernel and running all cells will reset the current working directory however will still return the desired imports/results.

Aloft

This not a good solution. %cd .. will give you a different result each time you run the cell (moves the working directory further up a level) – Siblee

@Siblee is there a reason why the cell containing this needs to be run multiple times? Maybe it's not compatible with your workflow but I can run this once at the top of my notebook and not have to worry about it again -- If I ever need to rerun the cells, restarting the kernel and running all cells gives the same result every time (moving the working directory up one) – Aloft

If needing to restart the kernel to rerun a notebook isn’t a code smell, I don’t know what is. How do other users of the notebook know that they need to do this? They just see import errors where previously there were none. – Siblee

If the magic command is in its own cell and the user isn't restarting the whole notebook, why would that cell even be rerun (assuming this magic command is the only thing in the cell)? Maybe this method doesn't work with everyone's workflows, but it helped me and I wish I would have been able to find a simple solution like this earlier. – Aloft

Im not sure i would call restarting the kernel a code smell in data science. I feel like its fairly common in EDA to go down one (or 11) paths that don't work out and leave you a pile of spaghetti. Restart-cleanup-continue on. – Heliogravure

Answer 4

I've solved this problem in the past by creating a symbolic link in the directory where the Jupyter notebook is located to the library it wants to load, so that python behaves as if the module is in the correct path. So for the example above, you would run the following command once per directory inside a Jupyter cell:

!ln -s /user/project/functions.py functions.py

and then you could import with

import functions

Note: I've only tried this on Linux and Mac Os, so I can't vouch for Windows.

Weepy

Answer 5

I would suggest to install functions.py as a package in your virtual environment. There are some benefits of this:

1. You can access functions.py file from any iPython notebook located in any place, but at the given environment (kernel).
2. Once you changed any function in functions.py file, you don't need to reload your iPython notebook again and again. It will automatically reload every change.

This is the way how it can be done:

• Create setup.py file (https://docs.python.org/2/distutils/setupscript.html) in your project folder
• Activate your virtual environment, go to your project location, and use this command pip install -e .

• Then, in your iPython notebook:

  %load_ext autoreload

  %autoreload 1

  %aimport yourproject.functions

  from functions import *

That's it!

Alfano

How would you share your work with this setup? I mean if coworkers need to replicate your env how is the best practice? – Ablaut

Answer 6

In addition to the answer from adhg, I recommend using Pathlib, for compatibility between Linux/Windows/WSL paths formats:

Assuming the following folder structure:

.
├── work
|   ├── notebook.ipynb
|   └── my_python_file.py
├── py
|   ├──modules
|   |    ├──__init__.py # empty
|   |    └──preparations.py
|   ├──__init__.py # empty
|   └── tools.py
├──.git
└── README.md

To load tools.py or preparations.py in my_python_file.py (or in notebook notebook.ipynb):

import sys
from pathlib import Path

# in jupyter (lab / notebook), based on notebook path
module_path = str(Path.cwd().parents[0] / "py")
# in standard python
module_path = str(Path.cwd(__file__).parents[0] / "py")

if module_path not in sys.path:
    sys.path.append(module_path)

from modules import preparations
import tools
...

Pantsuit

Answer 7

You can also use run magic in a jupyter notebook.

%run -i "<filepath/functions.py>"

Which basically runs the same as

from functions.py import *

Universe

Answer 8

Found myself in the same exact situation as the OP, going to create several notebooks hence the wish to organise them in different subfolders

Tried this that seems to do what I need and seems cleaner to me

import os
os.chdir(os.path.dirname(os.path.dirname(os.getcwd())))

My function being two levels above so nested two os.path.dirname (with different folder structure could be only one or more)

Just implemented it and working fine, and btw I'm using JupyterLab started... two levels above where the function resides

Finnigan

well, I just stumbled in one good reason not to, i.e. rerunning the cell. Avoidable using if os.getcwd().split('\\').pop() == 'definitions': os.chdir(os.path.dirname(os.path.dirname(os.getcwd()))) instead ('definitions' is my subfolder's name) – Finnigan

Answer 9

Another solution is to change the IPython kernel PYTHONPATH environment variable. The advantage of this solution, is that if you are always working with the same kernel, you don't have to add an

This solution is based on https://mcmap.net/q/118572/-how-to-set-env-variable-in-jupyter-notebook

Let's say that the IPython kernel that you use in your notebook is called venv1.

To find the kernel.json file associated with it, run in the terminal:

jupyter kernelspec list

you will get an output similar to:

Available kernels:
  python3                 <path1>/share/jupyter/kernels/python3
  venv1    <path2>/.local/share/jupyter/kernels/venv1
  python2                 /usr/local/share/jupyter/kernels/python2

Now, in the path associated with venv1, you will find the kernel.json file. open it with a text editor. for example, you can open it with gedit:

gedit <path2>/.local/share/jupyter/kernels/venv1/kernel.json

you should find a file similar to this:

{
 "argv": [
  "/homes/<user>/.conda/envs/venv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (venv1)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

You can set environment variables by adding a new dictionary with the key "env" (https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs), so we want to add to the PYTHONPATH the directory which you want to import the modules from, in your example this should be /user/project, so add change the file to look like this (don't forget the ',' before "env"):

{
 "argv": [
  "/homes/shaigu/.conda/envs/panoptic_match_gtoii/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (panoptic_match_gtoii)",
 "language": "python",
 "metadata": {
  "debugger": true
 },
 "env": {"PYTHONPATH": "${PYTHONPATH}:/user/project"}
}

Antoinette

Answer 10

Add parent directory to sys path:

import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(os.path.abspath(''))))

Hysteresis