Why do many Common Lisp systems use a single packages.lisp file?
Asked Answered
A

1

9

Many of the commonly used libraries I see use a single packages.lisp file to declare all of the packages in the library (system) in one place.

Since exported symbols are part of the package definition, this means individual source files don't list their exported symbols.

In my own projects, I prefer the style of defining one source file per package, and defining its interface/exports at the top of the file.

I am wondering if I am doing it wrong, or missing an essential concept that leads to the preference for a single packages.lisp file.

In case it's relevant, I'm also using ASDF's :package-inferred-system approach, and uiop:define-package instead of defpackage, to make use of it's handy symbol-shadowing :mix feature -- because I haven't figured out how to :use a package which shadows built-in symbols without re-declaring the shadows in each package that uses it.

Afresh answered 11/8, 2017 at 19:13 Comment(1)
I think the reason is mostly historical - a holdover from the pre-defpackage days when one had to jump through some weird hoops to make sure the package existed at file compilation time.Melancholia
A
20

Traditional view of using Packages

Sometimes a package (a symbol namespace) is used in tens or even hundreds of files and packages might need to be computed. This might be simpler in one file and it might be easier to get a textual overview when the package declaration(s) are in one file. For example the implementation of a text editor could be in just one package using around 100 files. Notice also that a package is really a runtime data structure in Common Lisp with a programmer interface.

Influence from other languages like Java

The style to have one package and one corresponding source file is often coming from outside of Common Lisp, from languages which usually have a correspondence of something like one class = one namespace = one file. This leads to a multitude of files, in nested directories, with often small pieces of code per file.

Common Lisp does not have these restrictions: methods are not organized in classes, classes are not namespaces, files can have any mix of definitions, ... The Lisp libraries tend to have large packages/namespaces and large files.

Structure and its limitations

Packages

Packages in Common Lisp are namespaces for symbols. This is not a full-blown module system. Also there is no actual information hiding. There is a distinction between symbols and exported symbols. Note also that symbols have multiple meanings as a name (variable, function/macro/special operator, class name, slot name, type, data object, package name, ...) and exporting a symbol from a package does not make a distinction between those meanings. Note also that it is possible, but not recommended, to use more than one package namespace in a file (for example by using multiple in-package forms). Typically one would have only one namespace in a file. This is usual declared by an in-package somewhere at the top of the file.

Classes

Classes are no namespace. Typically classes are defined using the built-in Common Lisp Object System, called CLOS. They bundle slots and are used for dispatching in CLOS generic functions. But they don't contain their methods.

Systems

Systems are not a language construct. They are an extension to Common Lisp. The idea of a system is provided by tools like ASDF. A system is a tool for organizing a bunch of files which make a library/application and their dependencies. It's also where functionality is provided to use actions on a system (compiling, loading, delivering, ...).

???

There might be something missing for better organization of code. Each project might need a slightly different way.

I would typically use files to put related functionality into one file and set up a bunch of files for system - if needed. But that could mean that a file implements more than one class and a lot of functionality. I tend to organize files in sections where directly related code is implemented. I might describe a few elements (classes, functions) at the top of the file, but that would be more of a local overview and less than a list of exported symbols. Once you load a system and its files into a Lisp with its IDE, it is the purpose of the development environment to let me query for code (where is? who uses? what is used? sub/super class? package contents? ...) and to provide browsers for that.

There have alternative ways to organize code. For example by using PROVIDE and REQUIRE, which are only very lightly described in the language standard. These tend to pull in functionality on demand and create package structures on the go.

There might be the need for something like object-oriented protocols, which provide more structure for CLOS.

Early use of packages and systems in the MIT Lisp Machine operating system

One of the early demands for packages to avoid name clashes and systems to organize code came from the Lisp Machine operating system developed at MIT in the late 70s and the 80s. It was developed in Lisp Machine Lisp and later in Common Lisp. There, a lot of different functionality (compiler, editor, listener, inspector, mail client, graphics library, file system, serial interface, ethernet interface, networking code, ...) was running in a single address space with probably tens of thousands of symbols. The package declaration was usually done in the corresponding system description file (here we again use the Lisp meaning of a system as a library or an application, a collection of files for a certain purpose) or sometimes in a separate file. Packages provided the namespace for large libraries or even entire applications. Thus files, packages, systems and even classes (earlier called Flavors) were already used then to structure software implementations. See the Lisp Machine Manual (version from 1984), Chapter 29, Maintaining Large Systems and Kent Pitman's MIT AI Memo 801 from 1984: The Description of Large Systems. Systems on the Lisp Machine had versions and supported patching (incremental versioned changes). Files had versions, too.

Augie answered 11/8, 2017 at 21:10 Comment(4)
This is very helpful and informative background. Can I safely conclude that using a one-package-one-file approach today is simply a matter of choice and there are no negative consequences (as far as Lisp, ASDF, Quicklisp etc) to not providing a packages.lisp file?Afresh
@anticrisis: it's just that it does not fit well into the usual Lisp style. There is very little to no advantage of using lots of little files with a namespace per file. In the end it is more confusing than using larger packages. Since Lisp does not promote strict information hiding, there is very little point to emulate the class=namespace=file style of programming in Lisp.Augie
I see -- I am wondering how to reconcile the idea of a single, global namespace spanning a hundred files with modularity, separation of concerns, subsystems with clear external APIs, etc. But I don't think you're proposing that extreme, either. As usual, it seems there is a lot of power in Common Lisp, and it's up to the team/user to design a system that works best for their needs.Afresh
The packages.lisp file often provides a good overview of a system too. You can see from these examples that symbols are grouped logically and it is very easy to see package nicknames, imports, shadows and shadowing imports.Washrag

© 2022 - 2024 — McMap. All rights reserved.