Jupyter and Common Lisp

Question

I'm trying to install cl-jupyter (A common-lisp kernel for Jupyter), and I can't make it work: when I open a new lisp notebook (or change the kernel of an existing notebook), it crashes after displaying the following messages:

[I 18:26:50.855 NotebookApp] Kernel started: ccba815a-9065-4fad-9d95-06f6291136d2
To load "cl-jupyter":
  Load 1 ASDF system:
    cl-jupyter
; Loading "cl-jupyter"
...............

cl-jupyter: an enhanced interactive Common Lisp REPL
(Version 0.7 - Jupyter protocol v.5.0)
--> (C) 2014-2015 Frederic Peschanski (cf. LICENSE)

kernel configuration = ((hb_port . 58864) (shell_port . 37462)
                        (transport . tcp) (iopub_port . 43232)
                        (signature_scheme . hmac-sha256)     (control_port . 52184)
                        (stdin_port . 45879)
                        (key . 2ae7d65f-65f9-40d8-bfd4-21760eaec0ca)
                        (ip . 127.0.0.1))
[Hearbeat] starting...
[Heartbeat] thread started
[Heartbeat] thread started
[Kernel] Entering mainloop ...
[Shell] loop started
Argh! corrupted error depth, halting
fatal error encountered in SBCL pid 24605(tid 140737353922304):
%PRIMITIVE HALT called; the party is over.

Some details:

• Jupyter works fine with python (both 2x and 3x).
• sbcl also seems to work fine (I'm just starting with common-lisp, so I might be missing something - but I can run code, and use asdf and quicklisp).
• I installed it by running "python3 ./install-cl-jupyter.py".
• My .sbclrc file now contains just "(load "~/quicklisp/setup.lisp")".

System Information:

• OS: Linux 4.1.13-1-MANJARO x86_64 GNU/Linux
• Python 3.5.0
• SBCL 1.3.0
• Jupyter version 4.0.6

Some things I've tried:

The file ~/.ipython/kernels/lisp/kernel.json contains:

{"argv": ["sbcl", "--non-interactive", "--load", "/home/myusername/lisp/systems/cl-jupyter/cl-jupyter.lisp", "/homemyusername/lisp/systems/cl-jupyter/src", "/home/myusername/lisp/systems/cl-jupyter", "{connection_file}"], "display_name": "SBCL Lisp", "language": "lisp"}

All the paths are correct. When I run the same code manually:

sbcl --non-interactive --load "/home/myusername/lisp/systems/cl-jupyter/cl-jupyter.lisp" "/homemyusername/lisp/systems/cl-jupyter/src" "/home/myusername/lisp/systems/cl-jupyter" kernel-07e04903-c562-4c67-bcc1-b68f4047d8d2.json

(where "kernel-07e04903-c562-4c67-bcc1-b68f4047d8d2.json" is an auto-generated file for a lisp-kernel that I saved), it "looks ok" (e.g. it runs, and waits with no errors).

I tried to do some quick debug-printing, and my best guess is that it crashes in src/shell.lisp, in the line

 (vbinds (identities sig msg buffers)  (message-recv (shell-socket shell))

But again, I could be wrong. Since it kinda looks like it has something to do with communication, I made sure that zeromq is updated.

I'm not sure what's next besides learning some more lisp, and then come back to properly debug the code - but I was hoping to learn it using Jupyter :)

I'd appreciate any suggestions, Thanks.

Dispersion

Can you uncomment the format call in message-recv at the end of src/message.lisp ? If it's not printed then it's probably some issue with zeromq. You can also try to trace some functions, to narrow down where exactly the error occurs. – Dramaturgy

This is internal to sbcl. One of *current-error-depth* or *maximum-error-depth* is not bound or not a realp number. See: cs.utexas.edu/users/jared/Milawa/Support/sbcl/sbcl-1.0.29/src/… (though note that this is from a different version of sbcl) – Dramaturgy

I'd guess that somehow !COLD-INIT isn't called: github.com/sbcl/sbcl/blob/… – Dramaturgy

This could be related to having multiple threads, as from this document I'd guess that it can happen in a multi threaded situation that *maximum-error-depth* is unbound: github.com/sbcl/sbcl/blob/… – Dramaturgy

So, to be precise, I think this is some threading related bug in SBCL. I modified the version of sbcl you're using to be more precise when reporting that error, i.e. to report whether it's really *maximum-error-depth* that is unbound. Could you try to run with this modified version of sbcl, in order to confirm my guess? The code is to be found there: github.com/musteresel/sbcl/tree/… The only modification is in src/code/cold-error.lisp to split the case checking for corrupted error depth (which you can test via github if you don't trust me ;) ) – Dramaturgy

Thanks! I compiled the source with your modification, and the error I now get is "Argh! corrupted maximum error depth, halting". So it looks like you're right. Do you have any idea how to solve this? – Dispersion

By the way, the line "(format t "~S" SB-KERNEL:*MAXIMUM-ERROR-DEPTH*)" prints 10 (as expected), either if I put it in shell.lisp just before printing "loop started" or if I put it in kernel.lisp just before printing "starting...." or printing "loop started". – Dispersion

And " (format t "~S" (SB-INT:FIXNUMP SB-KERNEL:*MAXIMUM-ERROR-DEPTH*))" prints "T"... – Dispersion

Answer 1

This issue is caused by a change in the size of the message structure, zmq_msg_t, in 0MQ version 4.1.x to 64 bytes from a size of 32 bytes in version 4.0.x.

The 0MQ interface library used by cl-jupyter is pzmq, and that library is stated to support 0MQ up to version 4.0. At the time of this writing, pzmq uses a 0MQ message size of 32 bytes. This leads to internal errors if 0MQ version 4.1.x is installed.

There are two solutions here:

1. Downgrade the 0MQ installation to version 4.0.x
2. Fix the pzmq library by updating the definition of msg% in c-api.lisp from 32 to 64 bytes.

Update - 9 May 2016

The pzmq library was updated to support 0MQ version 4.1.x, (preserving compatibility with older versions). The new version of pzmq is planned for release as part of the May 2016 Quicklisp release.

Update - 1 June 2016

The May 2016 Quicklisp release includes updates to the pzmq library which resolve this issue. A downgrade of 0MQ or modifications to the pzmq library are no longer necessary.

Uproot