Django shell_plus: How to access Jupyter notebook in Docker Container

Question

I am trying to access a Jupyter Notebook created with the shell_plus command from django-extensions in a Docker container.

docker-compose -f local.yml run --rm django python manage.py shell_plus --notebook

My configuration is based on the answers of @RobM and @Mark Chackerian to this Stack Overflow question. I.e. I installed and configured a custom kernel and my Django apps config file has the constant NOTEBOOK_ARGUMENTS set to:

NOTEBOOK_ARGUMENTS = [
    '--ip', '0.0.0.0',
    '--port', '8888',
    '--allow-root',
    '--no-browser',
]

I can see the container starting successfully in the logs:

[I 12:58:54.877 NotebookApp] The Jupyter Notebook is running at:
[I 12:58:54.877 NotebookApp] http://10d56bab37fc:8888/?token=b2678617ff4dcac7245d236b6302e57ba83a71cb6ea558c6
[I 12:58:54.877 NotebookApp]  or http://127.0.0.1:8888/?token=b2678617ff4dcac7245d236b6302e57ba83a71cb6ea558c6

But I can't open the url. I have forwarded the port 8888 in my docker-compose, tried to use localhost instead of 127.0.0.1 and also tried to use the containers IP w/o success.

It feels like I am missing the obvious here … Any help is appreciated.

Tacket

Answer 1

For the sake of records as of 2020, I managed to have a working django setup with Postgresql in docker-compose:

development.py (settings.py)

INSTALLED_APPS += [
    "django_extensions",
]

SHELL_PLUS = "ipython"

SHELL_PLUS_PRINT_SQL = True

NOTEBOOK_ARGUMENTS = [
    "--ip",
    "0.0.0.0",
    "--port",
    "8888",
    "--allow-root",
    "--no-browser",
]

IPYTHON_ARGUMENTS = [
    "--ext",
    "django_extensions.management.notebook_extension",
    "--debug",
]

IPYTHON_KERNEL_DISPLAY_NAME = "Django Shell-Plus"

SHELL_PLUS_POST_IMPORTS = [ # extra things to import in notebook
    ("module1.submodule", ("func1", "func2", "class1", "etc")),
    ("module2.submodule", ("func1", "func2", "class1", "etc"))

]

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true" # only use in development 

requirements.txt

django-extensions
jupyter
notebook
Werkzeug  # needed for runserver_plus
...

docker-compose.yml

version: "3"

services:
  db:
    image: postgres:13
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
    restart: always
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data/
  web:
    build: .
    environment:
      - DJANGO_SETTINGS_MODULE=settings.development
    command:
      - scripts/startup.sh
    volumes:
      - ...
    ports:
      - "8000:8000" # webserver 
      - "8888:8888" # ipython notebook
    depends_on:
      - db

volumes:
  postgres_data:

From your host terminal run this command:

docker-compose exec web python manage.py shell_plus --notebook

Finally navigate to http://localhost:8888/?token=<xxxx> in the web browser of host.

Seleta

The essential parts are NOTEBOOK_ARGUMENTS and to have port 8888 mapped outside from the container (-p param for example). – Fatty

@Fatty async part is also needed for db statements – Seleta

@Fatty needed for django 3.0 – Seleta

Instead of setting DJANGO_ALLOW_ASYNC_UNSAFE in settings.py, I recommend moving it to docker-compose exec -e DJANGO_ALLOW_ASYNC_UNSAFE=true web python manage.py shell_plus --notebook – Allpowerful

Answer 2

Got it to work, but why it does so is beyond me. Exposing the ports in the docker-compose run command did the trick.

docker-compose -f local.yml run --rm -p 8888:8888 django python manage.py shell_plus --notebook

I was under the impression exposing ports in my local.yml would open them also in containers started by run.

Tacket

I have provided a MWE of docker-compose, django, notebook. – Seleta

Thx. Gonna check it out on Monday. The weird thing with my solution was, that albeit the notebook would start and be accessible in the browser, none of the shell plus functionality was available, i. e. Module loading and stuff. I couldn't load any models of my app actually. – Tacket

run will spin up a new container, I think you should use exec. All the shell_plus functionality is accessible in the provided solution. – Seleta

Ok. It seems to work, although I can’t log into the notebook. The token won't be accepted. Which seems weird, all I am doing is to navigate to the token enriched url. You know why that might be? ... I tried to also enter the token manually, of course. – Tacket

Did a restart and the notebook starts now. But I got the same AppRegistryNotReady: Apps aren't loaded yet. error like before with my run solution. – Tacket

I guess I know what's going on. I am also using Wagtail on top of Django and I think that’s the culprit. Running your solution on a vanilla Django works. Therefore I'll accept your answer ;-) – Tacket

I am also using wagtail and it works fine. Though, no idea about the error you receive. – Seleta

Ah, really? Ok, at least I know it's possible. May I ask if you use an ClusterableModel in your models? I suspect that one to be the offender… – Tacket

No I don't have such kind of models – Seleta

Answer 3

The compose run command will per default not expose the defined service ports. From the documentation at https://docs.docker.com/compose/reference/run/

The [...] difference is that the docker-compose run command does not create any of the ports specified in the service configuration. This prevents port collisions with already-open ports. If you do want the service’s ports to be created and mapped to the host, specify the --service-ports flag:

docker-compose run --service-ports web python manage.py shell 

You will therefore need to run

docker-compose -f local.yml run --rm --service-ports django python manage.py shell_plus --notebook

It might also be that the default 8888 port is already used by a local jupyter server (e.g. one spun up by VS Code's jupyter notebook implementation. I therefore usually map to a different port in the settings.py NOTEBOOK_ARGUMENTS list. (In this case the port mapping in the compose file needs to be adjusted as well, of course, and there must not be another container running in the background with the same service definition as this might also occupy the port.)

Markettamarkey

Answer 4

If you want to use jupyter notebook like separated service:

jupyter_notebook:
  build:
    context: .
    dockerfile: docker/dev/web/Dockerfile
  command: python manage.py shell_plus --notebook
  depends_on:
    - web
  ports:
  - 8888:8888 # ipython notebook
  env_file:
    - .env

after:

docker-compose logs -f 'jupyter_notebook'

and you will get access token in logs

Lettered

Hi bandirom. Please explain your answer a bit more. What should we do with the YAML provided in your answer? Why does doing this help to open the URL that the original poster is trying to open? It looks like the original poster already has the access token, but can't open the URL. – Katz

Hi @SethDifley! You can create separated instance for your notebook in docker-compose.yml like service. – Lettered

In answer above, you shoud start jupyter manually. In this solution, you have a runned notebook already – Lettered