Connecting to a remote IPython instance
Asked Answered
A

5

38

I would like to run an IPython instance on one machine and connect to it (over LAN) from a different process (to run some python commands). I understand that it is possible with zmq : http://ipython.org/ipython-doc/dev/development/ipythonzmq.html .

However, I can not find documentation on how to do it and whether it is even possible yet.

Any help would be appreciated!


EDIT

I would like to be able to connect to IPython kernel instance and send it python commands. However, this should not be done via a graphic tool (qtconsole) , but I want to be able to connect to that kernel instance from within a different python script...

e.g.

external.py

somehow_connect_to_ipython_kernel_instance
instance.run_command("a=6")
Arris answered 2/4, 2012 at 13:33 Comment(1)
Related: #18147058Dateline
P
32

If you want to run code in a kernel from another Python program, the easiest way is to connect a BlockingKernelManager. The best example of this right now is Paul Ivanov's vim-ipython client, or IPython's own terminal client.

The gist:

  • ipython kernels write JSON connection files, in IPYTHONDIR/profile_<name>/security/kernel-<id>.json, which contain information necessary for various clients to connect and execute code.
  • KernelManagers are the objects that are used to communicate with kernels (execute code, receive results, etc.). *

A working example:

In a shell, do ipython kernel (or ipython qtconsole, if you want to share a kernel with an already running GUI):

$> ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-6759.json

This wrote the 'kernel-6759.json' file

Then you can run this Python snippet to connect a KernelManager, and run some code:

from IPython.lib.kernel import find_connection_file
from IPython.zmq.blockingkernelmanager import BlockingKernelManager

# this is a helper method for turning a fraction of a connection-file name
# into a full path.  If you already know the full path, you can just use that
cf = find_connection_file('6759')

km = BlockingKernelManager(connection_file=cf)
# load connection info and init communication
km.load_connection_file()
km.start_channels()

def run_cell(km, code):
    # now we can run code.  This is done on the shell channel
    shell = km.shell_channel
    print
    print "running:"
    print code

    # execution is immediate and async, returning a UUID
    msg_id = shell.execute(code)
    # get_msg can block for a reply
    reply = shell.get_msg()

    status = reply['content']['status']
    if status == 'ok':
        print 'succeeded!'
    elif status == 'error':
        print 'failed!'
        for line in reply['content']['traceback']:
            print line

run_cell(km, 'a=5')
run_cell(km, 'b=0')
run_cell(km, 'c=a/b')

The output of a run:

running:
a=5
succeeded!

running:
b=0
succeeded!

running:
c=a/b
failed!
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/Users/minrk/<ipython-input-11-fb3f79bd285b> in <module>()
----> 1 c=a/b

ZeroDivisionError: integer division or modulo by zero

see the message spec for more information on how to interpret the reply. If relevant, stdout/err and display data will come over km.iopub_channel, and you can use the msg_id returned by shell.execute() to associate output with a given execution.

PS: I apologize for the quality of the documentation of these new features. We have a lot of writing to do.

Phocis answered 2/4, 2012 at 15:51 Comment(4)
Thank you very much for the excellent answer! Exactly was I was looking for. Is it possible to receive the output of the command that ran in the kernel? e.g.: "a=2".."print a", and then receive "2"?Arris
And is it possible to call IPython.frontend.terminal.embed.InteractiveShellEmbed() to open as kernel?Arris
output comes over the iopub_channel. As I linked above, the message spec details these messages, and vim-ipython and ipython console demonstrate how to deal with these messages. As for the embed, I think you are asking for this addition, which should be merged before long.Phocis
This worked great for me with IPython 2.2.0, I just needed to replace BlockingKernelManager with IPython.kernel.blocking.client.BlockingKernelClient and the same method calls work. Thanks!Fayth
W
31

If you just want to connect interactively, you can use SSH forwarding. I didn't find this documented anywhere on Stack Overflow yet, yet this question comes closest. This answer has been tested on Ipython 0.13. I got the information from this blog post.

  1. Run ipython kernel on the remote machine:

    user@remote:~$ ipython3 kernel
    [IPKernelApp] To connect another client to this kernel, use:
    [IPKernelApp] --existing kernel-25333.json
    
  2. Look at the kernel-25333.json file:

    user@remote:~$ cat ~/.ipython/profile_default/security/kernel-25333.json 
    {
      "stdin_port": 54985, 
      "ip": "127.0.0.1", 
      "hb_port": 50266, 
      "key": "da9c7ae2-02aa-47d4-8e67-e6153eb15366", 
      "shell_port": 50378, 
      "iopub_port": 49981
    }
    
  3. Set up port-forwarding on the local machine:

    user@local:~$ ssh user@remote -f -N -L 54985:127.0.0.1:54985
    user@local:~$ ssh user@remote -f -N -L 50266:127.0.0.1:50266
    user@local:~$ ssh user@remote -f -N -L 50378:127.0.0.1:50378
    user@local:~$ ssh user@remote -f -N -L 49981:127.0.0.1:49981
    
  4. Copy the kernel-25333.json file to the local machine:

    user@local:~$ rsync -av user@remote:.ipython/profile_default/security/kernel-25333.json ~/.ipython/profile_default/security/kernel-25333.json
    
  5. Run ipython on the local machine using the new kernel:

    user@local:~$ ipython3 console --existing kernel-25333.json
    Python 3.2.3 (default, Oct 19 2012, 19:53:16)
    Type "copyright", "credits" or "license" for more information.
    
    IPython 0.13.1.rc2 -- An enhanced Interactive Python.
    ?         -> Introduction and overview of IPython's features.
    %quickref -> Quick reference.
    help      -> Python's own help system.
    object?   -> Details about 'object', use 'object??' for extra details.
    
    
    In [1]: import socket; print(socket.gethostname())
    remote
    
Whittington answered 7/3, 2013 at 10:39 Comment(3)
In case anyone doesn't know where the json file is located (like I didn't): it can be found by running jupyter --runtime-dirBenefaction
Exactly the answer I was looking for. Note that the -f flag could be used to specify the path of the JSON file.Urbai
if i understand correclty ipython console has been depreciated, use jupyter console --existing instead. See here: #22448072Louvenialouver
P
24

Update to minrk's answer after the split to jupyter. With jupyter_client (4.1.1) the simplest code is rather something like:

import jupyter_client

cf=jupyter_client.find_connection_file('6759')
km=jupyter_client.BlockingKernelClient(connection_file=cf)
km.load_connection_file()

km.execute('a=5')

Note that:

  • jupyter_client.BlockingKernelClient is also aliased with jupyter_client.client.BlockingKernelClient.
  • the shell (km.shell_channel) does not have the method execute() & get_msg() anymore.

Currently it is quite difficult to find an updated documentation; nothing yet on http://jupyter-client.readthedocs.org/en/latest/ for BlockingKernelClient. Some code in https://github.com/jupyter/jupyter_kernel_test. Any link welcome.

Peat answered 4/3, 2016 at 9:55 Comment(3)
Very nice answer!Threesome
Man that's amazing! It is both extremely simple and efficient. I like it!Yale
I'm in a JupyterLab notebook and two kernels are running. This code only connects to one of the kernels. How can I choose which one it connects to? Both have the same connection file.Pozzuoli
B
7

The above answers are a bit old. The solution for the latest version of ipython is much simpler but is not documented well at one place. So I thought I would document it here.

Solution to connect from any OS to a ipython kernel running on Windows

If either the client or server is a linux or other operating system, just change the location of kernel-1234.json appropriately based on Where is kernel-1234.json located in Jupyter under Windows?

  1. On your windows based kernel start, make sure ipykernel is installed using pip install ipykernel
  2. Start the ipykernel using ipython kernel -f kernel-1234.json
  3. Locate the kernel-1234.json file on your Windows machine. The file will probably have a different number, not 1234 and will most likely be located in 'C:\Users\me\AppData\Roaming\jupyter\runtime\kernel-1234.json': https://mcmap.net/q/410457/-connecting-to-a-windows-ipython-kernel-from-linux
  4. Install Jupyter Console (or Jupyter Qtconsole/notebook) using pip install jupyter-console or pip install qtconsole https://jupyter-console.readthedocs.io/en/latest/
  5. If you are on Windows do a ipconfig to find out the ip address of your Windows server. (On Linux do a ifconfig at the shell prompt). In the kernel-1234.json file change the ip address from 127.0.0.1 to the ip address of your server. If you are connecting from another Windows server, then copy the kernel-1234.json file to your local computer and note down the path.
  6. Navigate to the folder containing the kernel-1234.json and start Jupyter Console using jupyter console --existing kernel-1234.json
Boughpot answered 18/1, 2018 at 23:55 Comment(9)
In my Windows installation, I don't have ipykernel, instead I have jupyter-kernel. When this starts, it displays: Connection file: <path>\kernel-<guid>.json. In this file, it says "ip": "127.0.0.1", so I don't see how using this file on the client machine could be used on the other machine trying to use this kernel? When I change 127.0.0.1 to the IP address of the machine running jupyter-kernel (e.g. 10.0.0.122) the Jupyter Console repeats kernel died every 3 seconds.Saddlecloth
Yes. you are correct. When I wrote this, I was running the client and server on the same machine. In fact when you run Jupyter notebook, there is a client and a server which are running on the same machine, which is why 127.0.0.1 works fine. If you have to run your client and server on different machines, you need to 1. make sure that the server's ip is visible from the client. 2. in the kernel-1234.json, make sure to change the ip to whatever you client is set to.Boughpot
Not 100% sure, but the ip you mentioned 10.0.0.122 doesnt seem like a ip which would be visible from the client. Did you try pinging your server with ping 10.0.0.122 to check whether the server is visible from the client? and there is no firewall or something blocking the connection?Boughpot
What have you intalled on your server and on your client?Boughpot
Yes, the ping works. Both machines are on the same LAN (10.0.x.x and 192.168.x.x are common local LAN addresses). The original question specifically asked about accessing IPython on a machine on the LAN.Saddlecloth
Right now I am running the server and client in a Bridge Networking format so this does work on LAN connections. You will also notice there at multiple ports mentioned in the json file. Those ports need to be visible/open. Are you running any firewalls on your client and server?Boughpot
Also I had a typo, in the above solution, if you are starting the ipython kernel you need to start it with the same kernel-1234.json file. The kernel-1234.json file should have the same ip address.Boughpot
Can you try following the instructions using ipykernel, while using the file with the server ip and making sure there is no firewall running? When I do pip list on my server I don't see a jupyter-kernel. Also I don't see it listed as a kernel here: github.com/jupyter/jupyter/wiki/Jupyter-kernels. When I try pip install jupyter-kernel, it doesn't install anything. So I am not sure what you are running..Boughpot
Thanks, on the server, starting ipython kernel --ip 10.0.1.122 (to explicitly specify its IP address) causes the .json file it creates to have that IP in it, instead of 127.0.0.1), in addition it causes Windows Firewall to prompt to open the required ports. Then, on the client machine, starting the client using that same .json file works. It seems Python distributions are just starting to rename the IPython client parts to be Jupyter; my Anaconda dist had both. When I started ipython qtconsole on the client it warned I should start using jupyter qtconsole. Thanks a lot!Saddlecloth
H
1

If you're using Anaconda, in OS X the JSON file is stored at

/Users/[username]/Library/Jupyter/runtime/

In Windows:

c:\Users[username]\AppData\Roaming\jupyter\runtime\

Harriott answered 11/1, 2016 at 15:30 Comment(3)
... and c:\Users[username]\AppData\Roaming\jupyter\runtime\ on WindowsHarriott
Does this answer the question? The OP wants to "run an IPython instance on one machine and connect to it (over LAN) from a different process".Chartism
@WaiHaLee, I think it does provide a crucial piece of solving the question. For a beginner, the location of the kernel-1234.json file can be confusing. For full solution with the latest version of ipython/ipykernel: https://mcmap.net/q/404255/-connecting-to-a-remote-ipython-instanceBoughpot

© 2022 - 2024 — McMap. All rights reserved.