Jupyter starting a kernel in a docker container?

Question

I want to switch my notebook easily between different kernels. One use case is to quickly test a piece of code in tensorflow 2, 2.2, 2.3, and there are many similar use cases. However I prefer to define my environments as dockers these days, rather than as different (conda) environments.

Now I know that you can start jupyter in a container, but that it not what I want. I would like to just click Kernel > use kernel > TF 2.2 (docker), and let jupyter connect to a kernel running in this container.

Is something like that around? I have used livy to connect to remote spark kernels via ssh, so it feels like this should be possible.

Shanta

Answer 1

Full disclosure: I'm the author of Dockernel.

By using Dockernel

Put the following in a file called Dockerfile, in a separate directory.

FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel
CMD python -m ipykernel_launcher -f $DOCKERNEL_CONNECTION_FILE

Then issue the following commands:

docker build --tag my-docker-image /path/to/the/dockerfile/dir
pip install dockernel
dockernel install my-docker-image

You should now see "my-docker-image" option when creating a new notebook in Jupyter.

Manually

It is possible to do this kind of thing without much additional implementation/tooling, it just requires a bit of manual work:

1. Use the following Dockerfile:

FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel

2. Build the image using docker build --tag my-docker-image .

3. Create a directory for your kernelspec, e.g. ~/.local/share/jupyter/kernels/docker_test (%APPDATA%\jupyter\kernels\docker_test on Windows)

4. Put the following kernelspec into kernel.json file in the directory you created (Windows users might need to change argv a bit)

{
 "argv": [
  "/usr/bin/docker",
  "run",
  "--network=host",
  "-v",
  "{connection_file}:/connection-spec",
  "my-docker-image",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "/connection-spec"
 ],
 "display_name": "docker-test",
 "language": "python"
}

Jupyter should now be able spin up a container using the docker image specified above.

Sabol

It seems on Windows (running the linux docker) I run into this issue github.com/jupyter/notebook/issues/4998. Its just not entirely clear from the thread how I should fix it. Do you have this issue before? – Shanta

@Shanta it's because that file was somehow edited by the container, i.e. the kernel inside the container wrote there. Since it was binded to the container filesystem via docker daemon, which runs on admin privileges, the file got access permissions where only administrator can edit it. This is very setup specific. Does it occur at the first start of the kernel or after a retry? – Junker

It occurs when I start the kernel for the first time. I can get a different error if I use the ENV JUPYTER_RUNTIME_DIR=/tmp/runtime and adjust the location on the -f flag Timeout waiting for kernel_info reply. But if I also adjust the {connection_file}:/tmp/runtime/connection_spec I again get the error with permissions. No matter if I run jupyter as admin or not. – Shanta

@Shanta you shouldn't really adjust these parameters this way. The right side (after colon) of -v must be equal to the argument passed as -f. Check whether binding any file to container changes its permissions. Use docker run -it -v path\to\file:/test-file my-docker-image /bin/bash, and check permissions on path\to\file after that. The file must exist beforehand. Also, if you get a directory inside the container in place of the test-file, something is wrong. – Junker

Do you have any thoughts or pointers about how a similar thing can be achieved using singularity? HPC admins don't like to let docker run on the cluster as it requires elevated privilege. If it helps, singularity can natively run docker images. – Shirashirah

@Shirashirah no idea, I haven't used it. – Junker

@Shirashirah I realize my comment is a bit late, but it might be useful to someone. I just got this working for a kernel running with a singularity container. Basically, you replace the 'argv' entry in Michalik's answer above by sth like ["singularity", "exec", "path_to_sif", "python", "-m", "ipykernel_launcher", "-f", "{connection_file}"]. Cheers – Ringster

Answer 2

On Mac, the --network=host option doesn't work right (see https://docs.docker.com/network/drivers/host/) , requiring specific ports to be mapped. Building off of @Błażej Michalik's answer, I wrote the connect_docker.py script shown below.

Gist: modify the kernel.json to call my script instead of the kernel process. My script reads the connection file and starts up the kernel in the specified docker image with the necessary ports mapped and a modified connection file passed to the kernel in the docker image.

connect_docker.py

from pathlib import Path
import argparse
import json
import subprocess
import signal
import os


def main(connect_file: Path, docker_path, docker_tag, docker_run, docker_args):
    print('connection_file', connect_file)
    
    with open(connect_file) as file:
        connection_info = json.load(file)
    
    # Change host IP to "0.0.0.0" so port-forwarding works right
    connection_info['ip'] = '0.0.0.0'

    with open(connect_file, 'w') as file:
        json.dump(connection_info, file)

    port_maps = [
        tok
        for key in ['shell_port', 'iopub_port', 'stdin_port', 'hb_port']
        for tok in f'-p {connection_info[key]}:{connection_info[key]}'.split()
    ]
    
    container = f'kernel-{connection_info["key"]}'

    command = [docker_path, 'run', 
               '--rm',  # remove the container after it closes. Comment this out to debug a failed container.
               *port_maps, 
               *docker_args.split(),
               '--name', container,
               '-v', f'{connect_file}:/connection_file.json', 
               '-v', f'{os.getcwd()}:/notebook',  # mount current dir to /notebook
               '-w', '/notebook',  # and run in /notebook in the container
               docker_tag,
               *docker_run.split()
               ]
    print(' '.join(command))

    proc = subprocess.Popen(command)
    try:
        proc.wait()
    except KeyboardInterrupt:
        subprocess.run([docker_path, 'stop', container])
        raise


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("connection_file", help="Path to the ipykernel connection file", type=Path)
    parser.add_argument("docker_tag", help="Tag of the docker image to run in")
    parser.add_argument("docker_run", help="Command to run in docker (i.e. to start the kernel)",
                        default="python -m ipykernel_launcher -f /connection_file.json")
    parser.add_argument("--docker-args", help="Additional arguments to docker run", default="")
    parser.add_argument("--docker-exec", help="Path to the docker executable", default="/usr/bin/docker")
    args = parser.parse_args()

    main(args.connection_file, args.docker_exec, args.docker_tag, args.docker_run, args.docker_args)


""" Example Connection File (used for debugging this script during development)
{
  "shell_port": 63101,
  "iopub_port": 63102,
  "stdin_port": 63103,
  "control_port": 63106,
  "hb_port": 63105,
  "ip": "127.0.0.1",
  "key": "44ca1254-f8fb96abfcf5fc752e404e23",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": "py311-docker"
}
"""


"""

Here's an example kernel.json (fill in the ... with your relevant paths):

{
 "argv": [
  ".../python",
  ".../connect_docker.py",
  "--docker-exec",
  "/usr/local/bin/docker",
  "--docker-args",
  "--platform linux/amd64",
  "{connection_file}",
  "byubean/xeus-cling-kernel:winter2024",
  "micromamba run -n cling xcpp -f /connection_file.json -std=c++17"
 ],
 "display_name": "C++17 (docker)",
 "language": "C++17"
}

Ferromanganese