HOWTO definition and usage of Common Lisp packages (libraries)?

Question

I have developed some Common Lisp functions in a couple of Lisp source files that I'd like easily available to other functions I write, or make available on github if I think they'd be useful for someone else. For now, I've just been putting them in some pre-defined folder and using (require "/path/to/my/modules/module.lisp").

I'm wanting to understand what is the correct (canonical Lisp) way of defining Lisp library of modules. And the second part of the question is how to use them (whether I've defined them, or whether I've obtained one from someone else).

I've been reading a lot about defpackage and defsystem and asdf. But everything I've read seems to focus on some specific corner of the universe of this task. I'm having a lot of trouble seeing the big picture of custom module creation, deployment, and use. So assuming I have the Lisp environment in front of me (CLISP or SBCL) and one or more .lisp files I'd like to make a package or library out of, is there a document somewhere that explains what steps are required to do that? It's probably something I've already read, but didn't track due to not understanding the context. What I've read about ASDF seems functionally to be what I'm after, but I'm left not understanding whether ASDF is my only option, or whether it just happens to be a de facto standard and what most other people use, or whatever. I played with it a bit in SBCL and wasn't sure I was using it right, and didn't see info on how to set it up in CLISP. So I'm wanting to understand what is the up-the-middle, vanilla approach to this task.

I know this is a big, sloppy set of sub-questions. Again, if there are some good references to look at, I can read. I'm just having some trouble getting a big picture view of how this is supposed to work, and whether there is any "best" approach, or whether, in Lisp, it's a bit of a "Wild West" choose-the-library-manager-you like approach. I did the Google thing and read anything that looked relevant, but my brain is spinning from all of it.

Thanks.

Pushover

There is no accepted answer 6 years later. And I think I know the reason. None of the answers shows an example of what anyone new to LISP will probably be stuck on: Define a package in one file, then a) use that package from repl and b) use that package from another file. That makes like 5 lines of code and none of the very verbose tutorials and books on the web and answers here bother to show that. – Tibbetts

Answer 1

A system is a collection of files and sub-systems. One can compile or load such a system. There are also other operations possible. It keeps track of dependencies and tries to do a minimal amount of work.

If you are using SBCL and CLISP, then ASDF is the tool to choose. See http://www.cliki.net/asdf

ASDF provides, amongst other things, a DEFSYSTEM macro to describe such systems.

Don't use PROVIDE/ REQUIRE- unless you know what you are doing. ASDF is the way to go.

To publish your code and make it easily loadable by others then use QUICKLISP. See: http://www.quicklisp.org/beta/

Fere

Thanks Rainer. Are QUICKLISP and ASDF two different ways of doing the same thing, or do they work in conjunction? – Pushover

The latter. Quicklisp uses ASDF system definitions. – Acuna

@mbratch Also if you want an easy way to make quicklisp compatible projects I highly recommend quickproject. It will give you a nice template project with it's system and packages all set up. – Effect

Answer 2

A package in Common Lisp is not like a package in most other senses. It's not an archive of list files, but more like what most other languages would call a module or namespace that lets you select which symbols (names) from your code you want to show to the outside world of code.

If you just have one file for your little library, you can just distribute that. If you have multiple files, that where a tool like ASDF comes in to make sure, for example, that files defining macros are loaded before the files that use those macros.

Here are some good resources for you to look at, both chapters from Practical Common Lisp:

• 21. Programming in the Large: Packages and Symbols: This will give you a much better sense of what packages are and how and why to use them.
• 32. Conclusion: What's Next?: The "Delivering Applications" section has some good resources on this stuff in general, and has some mention of ASDF.

Foreside

Thanks! I'll check those links again. I read them before, but I wasn't getting the big picture. I should have clarified I did gather that "package" was more of a namespace definition, so I "abused" the term a bit in my question. I also encountered defsystem. My quandry is that I had trouble piecing it all together. Supposing I have a collection of Lisp "libraries", each in their own file, functionally not related, but may have some interdependecies (one may include another), is my scheme of a standard home for them and using require as needed a suitable approach? – Pushover

@mbratch: I'm not a very good authority on this, but I'd guess from experience in other languages that for your own use that is perfectly fine. If you are packaging your library for the use of others, though, you might want to look into ASDF. – Foreside

Answer 3

ASDF and Quicklisp are useful tools, that is an established fact. However, I would like to give an alternative point of view on the concept of "library" as it is discussed on previous answers

ASDF is designed to automate the compilation and loading of a set of source files. It is to CL what make is to Unix. It is perfectly valid to write and distribute a program without make just as it is fine to write a program in CL that does not use ASDF.

If your project is simple enough, it is sufficient to provide a file (e.g. load.lisp) that contains the statements to load the dependent files of your project in the right order. Therefore ASDF is not involved in the concept of a library.

In a provocative way, I would say that the canonical way of defining a module in CL is to use the defpackage declaration because it is the language unit that allows a programmer to isolate his/her declarations from those of someone else.

Then is the question of how to make it available to others. If you write portable CL code, then ASDF is the most popular system definition facility and you should use it. If you want to make it easier for others to obtain, then Quicklisp is the tool that changed the face of CL in the last few years.

Finally I would like to add that neither ASDF nor Quicklisp are standards, they are tools (which does not remove to their usefulness). ANSI Common Lisp is a standard and I would love to see a system definition standardized in CL.

Gelasius

Thanks for this (+1). Each answer given has helped fill in more puzzle pieces for me on my question. So if I start by looking at my own, private, library of lisp utilities, then having them as lisp files in a common folder somewhere and doing a (load ...) when I need them is a "canonical" way of doing it? – Pushover

I would not say "canonical" but "minimal", yes that's basically how you would do to keep things the leanest (and sometimes the best for learning). – Gelasius

Answer 4

Yes, ASDF is a de facto standard, and Quicklisp is another standard.

From Lisp's viewpoint what you want is to define one or several namespaces (packages). This is regulated by the ANSI standard. From your code's viewpoint you want to arrange a bunch of files so that they became a whole and somehow provide that packages. This is where ASDF plugs in. And Quicklisp allows you to manage ASDF systems in the easiest way concievable. You can both download a lot of libraries from Quicklisp's repository and manage your local systems creating symlinks in the quicklisp/local-project folder.

If you have Quicklisp installed, you can type

(ql:quickload :cl-fad)

at the REPL and thus load the CL-FAD library (possibly downloading it); then the packages CL-FAD and PATH become available, you forget about ASDF-systems and stick with the logic of packages.

A good idea would be to take a look at asd-files of several projects downloaded with Quicklisp.

Russom

I am biased, of course, but UIOP is vastly superior to CL-FAD for most (all?) of what CL-FAD does, i.e. provide portable access to the filesystem. And UIOP is vastly more portable, and more ubiquitous (part of ASDF 3). – Group

Answer 5

I setup my own libraries, that I use often, using ASDF system definitions and linking them into the local-projects (or similar named) folder of Quicklisp. That allows me to load them using quicklisp like any published package.

If you want to learn how to setup such an ASDF system, I would suggest installing Quicklisp (which for once is really easy, when it comes to installing 'unfinished' software from the internet), quickload a well known package or two and look at its .asd file, consulting the documentation when in doubt.

This way, you have your libraries already setup to be published like most of the well known CommonLisp packages out there.

Acuna