How do I add a kernel on a remote machine in IPython (Jupyter) Notebook?

Question

Dropdown menu in the top-right of the UI on a local machine (PC):

Kernel-> 
    Change kernel->
        Python 2 (on a local PC)
        Python 3 (on a local PC)
        My new kernel (on a remote PC)

Whitethorn

Answer 1

IPython use kernel is a file in ~/.ipython/kernel/<name> that describe how to launch a kernel. If you create your own kernel (remote, or whatever) it's up to you to have the program run the remote kernel and bind locally to the port the notebook is expected.

Ephah

Is there a fundamental reason the kernel must bind locally, or is it possible that we could someday see core support for remote kernels that bind to remote ports? – Annul

The current kernels are limited to what ZMQ can do. ZMQ can already can do remo host, though it's not advised; for security reason you should likely wrap the connection in another protocol if the network is not trusted. – Ephah

Answer 2

The IPython notebook talks to the kernels over predefined ports. To talk to a remote kernel, you just need to forward the ports to the remote machine as part of the kernel initialisation, the notebook doesn't care where the kernel is as long as it can talk to it.

You could either set up a wrapper script that gets called in the kernel spec file (https://ipython.org/ipython-doc/dev/development/kernels.html#kernel-specs) or use a module that can help you set up and manage different kinds of remote kernels: (pip install remote_ikernel; https://bitbucket.org/tdaff/remote_ikernel).

If you are using remote_ikernel, and have ssh access to the machine, the following command will set up the entry in the drop down list:

remote_ikernel manage --add \
    --kernel_cmd="ipython kernel -f {connection_file}" \
    --name="Remote Python" --interface=ssh \
    --host=my_remote_machine

Ardis

what is connection_file supposed to be here? The kernel-XXXX.json file that is generated on the remote machine with the relevant port information? And what is my_remote_machine supposed to be? – Ketchup

@Ketchup you don't need to change connection_file, that is part of the command that launches the kernel. Replace my_remote_machine with the hostname of the machine that you ssh to. – Ardis

Wow! Works like a charm! Thanks! – Decane

Any idea how to start a kernel in a specific virtual env? :) – Protractor

@Protractor you should be able to substitute ipython in the --kernel_cmdfor /full/path/to/venv/ipython and it will start the kernel there. You'll need to have the ipykernel packages installed within the virtual environment for that to work. – Ardis

@tdaff; thanks - but I still get the default env. See my question here #55731069 – Protractor

@Ardis Thanks! Is there any update after 5 years? Is remote_ikernel actively maintained and still the way to go? – Prescriptible

Answer 3

Remote jupyter kernel/kernels administration utility (the rk): https://github.com/korniichuk/rk

1. Install the rk from GitHub:

   $ sudo pip install git+git://github.com/korniichuk/rk#egg=rk

2. Setup SSH for auto login without a password:

   $ rk ssh

3. Install a template of a remote jupyter kernel:

   $ rk install-template

4. Change the kernel.json file:

   $ sudo gedit /usr/local/share/jupyter/kernels/template/kernel.json

For example from remote_username@remote_host to [email protected].

Click: Quickstart and YouTube video (less than 3 min).

Whitethorn

Answer 4

remote_ikernel in an earlier answer hasn't been updated in a while. It may still work fine, but as a more recent/updated option with some additional features, I've just tested ssh_ipykernel, and it works well for this. I'm using it to run a kernel on a remote machine with a GPU, connecting via SSH.

Leeke