Interact with a locally long-running Common Lisp image, possibly daemonized, from the command line

Question

How could one interact with a locally long-running Common Lisp image, possibly daemonized, from the command line?

I know it is possible to run a Common Lisp function from a Terminal command prompt, I am also aware of this.

I would need to do a similar thing, but with a local, already long-running Common Lisp image, being able to poll available functions from the CLI or shell scripts.

Is there a way to do that from a CLI, for example calling a function from a bash script, and receiving back whatever the function returns?

I though one could, for example, create a primitive web service, perhaps using woo or Hunchentoot, calling functions and fetching returned values via curl or wget, but it feels a little convoluted.

Of course, that is one of the many features of Emacs' SLIME, but I would need to call functions just from the CLI, without invoking Emacs.

Is there perhaps a way to reach a swank backend, outside of SLIME?

If possible at all, what would be the lisp idiomatic way of doing that?

I would be grateful for any pointers.

---

Update

Additional note

Many years ago, I was intrigued by being able to telnet into a long-running LISP image (I believe in this case uppercasing the name should be fine). If I remember correctly, it was available at prompt.franz.com. An article, somehow connected: telnet for remote access to a running application

Telnet is of course quite unsafe, but the usefulness of being able to access the Lisp application(s) in that way, for whatever reason, cannot be overstated, at least to some people.

Some additional pointers, and thanks

I would like to thank Basile Starynkevitch for his elaborate and thorough answer, especially on the theoretical aspect. I was looking for a more practical direction, specially connected to Common Lisp. Still, his answer is very instructive.

I was all ready to start writing a local server, perhaps using one of the fine Common Lisp libraries, like:

• usocket: Universal socket library for Common Lisp
• iolib: Common Lisp I/O library
• cl-aync: Asynchronous IO library for Common Lisp

But, thanks to Stanislav Kondratyev, I didn't have to. He pointed out an already existing solution that nicely answer my question, ScriptL: Shell scripting made Lisp-like

I tested it with success on Linux, FreeBSD and OS X, just make sure to install the thin wrapper over POSIX syscalls first. Among many features (see README), it allows exposition of just selected functions, security is properly handled, and it even supply a custom C client, which builds as part of the ASDF load operation, and supports a number of new features, such as I/O, in place of netcat.

Incision

Why do you ask? What is the context? Why do you want to avoid using emacs ? – Frenulum

Thanks for the comment. I have long-running processes running in the Lisp image, which handle data, also sitting in the lisp-image (ie, not written to disk or external database), and I need users to interact with that data from a CLI, without assuming the presence of SLIME, or EMACS – Incision

Have you considered using something like screen or tmux to start the image? – Honorable

Thank you, one could also use cl-daemonize (github.com/mishoo/cl-daemonize). That is not the issue, though. I need to communicate (call functions and fetch corresponding results) with the image. – Incision

Thanks for the pointer to cl-daemonize, which I did not know about. A read of the HOWTO on that page doesn't make it clear how you would interact with a daemonized image, other than sending it signals. With screen or tmux, you could just connect to the running process and use the REPL. – Honorable

swank-client is another related piece. It lets you talk to a swank server from another running common lisp repl. Hopefully some other method can raise this experience to terminal. – Firstfoot

Answer 1

You may find scriptl useful: http://quickdocs.org/scriptl/. However, it depends on iolib, which depends on some nonstandard C or C++ library, so building it is not quite straighforward.

It is indeed possible to communicate with a swank server if you familiarize yourself with the swank protocol, which seems to be underdocumented (see e. g. here: https://github.com/astine/swank-client/blob/master/swank-description.markdown). However, this exposes a TCP socket over a network, which could be unsafe. Once I tried that, too, but I was not satisfied with the IPC speed.

A while ago I wrote a rather naive SBCL-specific server that uses a local domain socket for communication, and a client in C. It's very raw, but you could take a look: https://github.com/quasus/lispserver. In particular, it supports interactive IO and exit codes. The server and the client form the core of a simple framework for deploying Unix style software. Feel free to borrow code and/or contact me for explanations, suggestions, etc.

Normally

Thank you very much. Your answer is very close to what I was looking for. I need a little bit to wrap my head around a few more options. Regarding a swank client, I was also considering using lime (github.com/eudoxia0/lime), by calling a proper lisp script from the cli, using that to connect to swank. Still, too convoluted. Thank you also for your kind availability. I will certainly be in touch. – Incision

Answer 2

It certainly is operating system specific, because you want some inter-process communication, and they are provided by the OS.

Let's assume you have a POSIX like OS, e.g. Linux.

Then you could set up a socket(7) or fifo(7) to which you send s-exprs to be evaluated. Of course you need to adapt the Common Lisp program to add such a REPL.

SBCL has some networking facilities, you could build on them.

Of course, you should understand first how IPC work on your OS. If it is Linux, you could read Advanced Linux Programming (it is centered on C programming, which is the low-level way of using OS services on POSIX, but you can adapt what you have learned to SBCL). And indeed, the Common Lisp standard might not have an official POSIX interface, so you need to dive into implementation specific details.

Perhaps you should learn more about BSD sockets. There are tons of tutorials on them. Then you could use TCP sockets (see tcp(7)) or Unix ones (see unix(7)). Advanced users could use the unsafe telnet command. Or you might make your software use SSL, or perhaps some libssh e.g. use ssh as their client.

You could decide and document that the protocol between user apps and your program is : send some-sexpr (on a documented socket) from user-app code to your server which is terminated by a double newline or by a form feed, and have your server eval it and send back the result or some error message. I did similar things in MELT and it is not a big deal. Be careful about buffering.

^{I guess that you assume that you have responsible and competent users (so don't open such a remote REPL to the wild Internet!). If you care about malicious or stupid use of a remote REPL, it is becoming complex.}

Alternatively, make your server a web application (by using some HTTP server library inside it), and ask users to use their browser or some HTTP client program (curl) or library (libcurl) to interact with it.

Frenulum

Thank you, I have added the OS detail. Would you care to add some details on how to add inter-process communication from the Lisp side, is there a good library (ie, at least partially documented)? – Incision

Thanks for the IPC note. I know about sockets, but I was hoping to find an easier path. cl-cookbook.sourceforge.net/sockets.html github.com/usocket/usocket – Incision

On Linux using sockets or fifos is probably the easiest, by adding the documented convention that users are expected to send a two-newlines terminated s-expr to be evaluated. Of course you need to manage the buffering. – Frenulum