How to use Common Lisp sort of like a smalltalk image

Question

Goal

I would like to have my Common Lisp (SBCL + GNU Emacs + Slime) environment be sort of like a Smalltalk image in that I want to have a big ball of mud of all my code organized in packages and preferably projects. In other words I have messed about a bit with save-lisp-and-die and setting Lisp in Emacs to bring up the saved image. Where I get lost is the appropriate way to make it work with Swank.

Problem

I believe it is required to put swank hooks inside my Lisp image before save-lisp-and-die. But it seems a bit fragile as on change to either my SBCL version or Slime version it seems to throw a version mismatch.

Question

Am I missing something? Do people work this way or tend to be more separate project as a loadable set of packages under ASDF?

I really miss the Smalltalk way and feel like per project ASDF is a bit clunkier and more rooted in the file system. In comparison it reminds me too much of every other language and their app/project orientation. OTOH it seem a bit more stable-ish re-versions of depended upon packages. Well, the entire versioning hell across languages is another matter.

Any hints how to do what I want or why it isn't such a good idea would be much appreciated.

Luthuli

Answer 1

Images

Common Lisp implementations like SBCL support images. The idea of saved memory appeared early in Lisp in the 60s.

Smalltalk took that idea from Lisp. In many Smalltalk implementations images might be portable (OS, runtime, ...) - especially when using machine independent byte code. SBCL OTOH compiles to native machine code.

Managed source code

Smalltalk added the idea of managed source code. Smalltalk often uses a simple database plus a change log to store source code. One Lisp doing something similar was Xerox Interlisp - but with slightly different approaches.

Other Lisp implementations / IDEs don't support managed source code that way - only the Xerox Interlisp variants - AFAIK.

DEFSYSTEM

In Common Lisp the use of defsystem facilities like ASDF and IDEs like GNU Emacs + SLIME is much more file system based. Code resides in multiple systems, which are files in a directory with a system description.

It's not even clear that it's meaningful to load a newer version of a system into a Lisp system where an older version is loaded. One might be able to arrange that, but there is nothing preventing me from messing that up.

Updating Lisp

Updating a Lisp like SBCL from one version to another might

• make the saved image incompatible to the runtime
• make the compiled code in FASL files incompatible with the runtime

You might save an image with the runtime included/bundled. That way you have the right combination of image and runtime.

But when you update the runtime, you usually/often need to regenerate a new compatible images with your code loaded.

Since SBCL brings releases once a month, there is a temptation to update regularly. Other implementations might use different strategies: LispWorks is an example. LispWorks is released much less often and publishes patches between releases, which are loaded into the released version.

Updating SLIME

I have no idea if it would be possible to update a loaded SLIME (a SLIME which has been already loaded in an earlier version into a Lisp system) by loading a new version on top. Probably a good idea to check with the SLIME maintainers.

Entrant

This is a good answer but I’d be curious to see how close one could get to a smalltalk-like system. I suppose the main challenges are threads (sbcl can only save when one thread is running so threads need to know how to save themselves and the main function needs to know how to restore them. There are some hooks but swank doesn’t use them), and os state like open files and other streams, and possibly dynamically loaded libraries (don’t really know how they’d be treated). – Snowclad

@DanRobertson: what does Smalltalk-like mean? More features for image saving/restoring? That depends on the implementation and its implementation strategy. Does it mean a managed source code database? Or does it mean the combination of both, which existed only for the commercial Interlisp-D / Medley system, – Entrant

well I suppose I mean “any combination of those features.” I certainly don’t know of any commonly used way to do any of the above in sbcl and your answer gives a good explanation of where such features have historically been in Lisp systems, how cl/sbcl are different and why such features would be difficult. When I wrote the above comment I guess I was hoping that someone else who saw this question might have some reference to some more modern attempt to implement some of those features in a modern cl. Perhaps such an attempt just does not exist. – Snowclad

@DanRobertson: LispWorks supports saving 'sessions': lispworks.com/documentation/lw71/IDE-M/html/… – Entrant

@DanRobertson just to clarify, do you mean that in SBCL when you restore an image after (sb-ext:save-lisp-and-die), you don't see previous threads restored? – Misconstruction

@Misconstruction it’s a complicated process as you can only save-Lisp-and-die if you are only running one thread – Snowclad

@DanRobertson: your're right! I just read the notes at sbcl.org/manual/#Saving-a-Core-Image. FWIW Lispworks isn't much better in this regard: lispworks.com/documentation/lw71/IDE-M/html/… states "All threads are killed before saving, so any data that is accessible only through a mp:process object, or by a dynamically bound variable, is not accessible." – Misconstruction

I can see an image after save-image-and-die. I have had more limited success getting it to load peacefully in my Slime/swank environment. For the record I am less interested (for now) in loading new versions of some projects than is simply having a long persistent session, possibly across more than one machine, without having to resort to some mega package.lisp to reload them all at startup. It would be nice if this 'just worked' across the several machines I run sbcl/slime on (laptop vs desktop and sometimes the laptop is OSX instead of linux). I wish for save image not so fragile. – Luthuli

Personally I have had the sneaking suspicion that file systems should go away since the 1980s. :) – Luthuli

Something that might be useful re the above is els.elsaa.org/static/2017/domkin.pdf Actual paper at vseloved.github.io/pdf/loading-multiple-versions.pdf – Luthuli

FWIW, I tend to go for a hybrid model when developing Common Lisp applications is to have a long-running image I treat pretty much as a big-ball-o-mud, but als make sure to (try to) get all my code into files in the file system, using something to get them loaded in the right order. Then, once in a while, I wash my big-ball-o-mud away and start from scratch. Because if I cannot start from scratch, I am SOL if I would want to run the native code on another processor architecture. – Worsham