How to serialize and load an object in SBCL/Common Lisp

Question

I have an object o that is an instance of a class X in SBCL.

I want a function write-X-object that serializes o to a file in such a way that when that file is read back in with load-X-object, the resulting object is equivalent to o.

;; writing the object
(write-X-object o "~/tmp/o.serialized")

;; reading the object, much later, 
;; after sbcl has been exited and restarted

(setq v (read-X-object "~/tmp/o.serialized"))

o might be about a gigabyte in size (or an array of several million smaller objects), with a complex structure, so the idea is for the reading and writing to be as quick as possible.

Merla

Check out github.com/conspack/cl-conspack – Ritchey

Answer 1

There are three main ways to do this

1. Use the built in facilities of print/read: you can define a method on print-object for your class that would serialise it (perhaps you would have it depend on some special variable so you don't print gigabytes to the repl). Then you could define a reader macro (corresponding to whatever syntax you use to print your object) then to save your object you would do (with-open-file (x "/tmp/foo" :direction :output) (print my-object X)) and to get it back you would do (with-open-file (x "/tmp/foo") (read x). The pros are that this is simple. The cons are that this is slow and is not space-efficient.
2. You could take advantage of a third-party serialisation library like conspack which a commenter suggested. Pros: reasonably fast to read or write. Cons: you can't incrementally read an object larger than memory.
3. You could restructure your class (using the MOP) so that the object can be stored in memory and on disk in the exact same format and then use mmap to read/write it. An example library for this is manardb. This system also allows lots of different objects to be stored. Pros: no overhead to read/write to/from disk. Can handle objects larger than main memory automatically (the is handles swapping in/out of ram for you). Cons: may be small overhead for accessing fields of object. Note that with this method, one can normally avoid most of the cons by using an implementation that can properly optimise access through ffi pointers and by using specialised data structures (e.g a float array being much better than a general array)

Englishry

Thanks. Just to be clear, I am trying to avoid option 1 for a several reasons: it's slow, it's space inefficient, and it's a hassle to code. I will look into option 2. – Merla

This is late but option 1 can be quite space efficient if the file is compressed (though this can be slow) although it has other drawbacks. – Englishry

Answer 2

like define a method print-object, I guess you can override generic-function MAKE-LOAD-FORM by defmethod.

make-load-form and make-load-form-saving-slots have defined in HyperSpec.

They return two forms, then you can write form to file with text, although that may take more space of disk (compare write specific binary spec).

---

And CLiki has a page about serialization. Some third-party package would be work at now I think.

Drais