How do I iterate through a directory in Common Lisp?

Question

I'm using OpenMCL on Darwin, and I'd like to do something like:

(loop for f in (directory "somedir")
  collect (some-per-file-processing f))

But I can't get directory to return anything other than NIL, and I can't seem to find any good explanation online (other than "its different for each system").

Any pointers?

Heterodox

Answer 1

Does your pathname specification contain a wildcard? Common Lisp's pathname stuff is somewhat hard to grasp at first - at least for me it was... As the CLHS states on the directory function:

If the pathspec is not wild, the resulting list will contain either zero or one elements.

In order to have your pathname include a wildcard, you might try the make-pathname function, like

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type "lisp"))

Or even

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type :wild))

I found the CL-FAD library a great help for dealing with pathnames and the file system. In particular, its list-directory function might be easier to use than the plain standard directory function.

Pangaro

Yep worked for me - (directory "pathname") returned NIL, where (directory "pathname/*.*") gave me expected results. – Heterodox

You need only files with names containing a dot? – Cosignatory

Weird huh? I'm actually after .h and .cpp files, but "pathname/*" returns NIL. – Heterodox

Might have something to do with the way Lisp merges "incomplete" the pathnames with some defaults. As your "/path/*" does only contain a wildcard for the name, but not for the type, I guess the CCL simply takes the type to look for from the default-pathname or something. I have been bitten often enough by the unwarranted assumption, that a pathname/pattern valid in a shell was also a valid/equivalent Lisp pathnames – Pangaro

Answer 2

There are basically two ways to specify pathnames:

Using strings

Strings are obviously depending on the platform: Unix syntax vs. Windows syntax for example.

"/Users/foo/bar.text"  is a valid pathname
"/Users/foo/*/foo.*"   is a valid pathname with two wildcards

You can create a pathname object from a string:

? (pathname "/Users/bar/foo.text")
#P"/Users/bar/foo.text"

The #p above assures that a pathname object (and not a string) is created, when you read it back.

? #P"/Users/bar/foo.text"
#P"/Users/bar/foo.text"

So, internally Common Lisp works with pathname objects, but it lets you use normal strings and makes pathname objects from them if needed.

When Common Lisp sees a pathname that has not all components specified (for example the directory is missing), then it fills in the components from the pathname object that is the value of the variable *DEFAULT-PATHNAME-DEFAULTS* .

With the function DESCRIBE you can look at the components of a pathname (here Clozure CL):

? (describe (pathname "/Users/bar/*.text"))
#P"/Users/bar/*.text"
Type: PATHNAME
Class: #<BUILT-IN-CLASS PATHNAME>
TYPE: (PATHNAME . #<CCL::CLASS-WRAPPER PATHNAME #x3000401D03BD>)
%PATHNAME-DIRECTORY: (:ABSOLUTE "Users" "bar")
%PATHNAME-NAME: :WILD
%PATHNAME-TYPE: "text"
%PHYSICAL-PATHNAME-VERSION: :NEWEST
%PHYSICAL-PATHNAME-DEVICE: NIL

Using the Lisp functions creating pathname objects

MAKE-PATHNAME is the function and it takes a few keyword arguments to specify the components.

Sometimes it is also useful to create a new pathname based on an existing one:

(make-pathname :name "foo" :defaults (pathname "/Users/bar/baz.text"))

If you use the function DIRECTORY it is useful to use a pathname with wildcards. DIRECTORY then will return a list of matching pathnames. The name DIRECTORY is slightly misleading, since DIRECTORY does not list the contents of a directory, but lists the matching pathnames for (usually) a pathname with wildcards. The wildcards can match a sequences of characters in components like /foo/s*c/list*.l*. There is also the wild card ** , which is used to match parts of a directory hierarchy like /foo/**/test.lisp , which matches all files test.lisp under the directory foo and its subdirectories.

(directory "/Users/foo/Lisp/**/*.lisp")

Above should return a list of all lisp files in /Users/foo/Lisp/ and all its subdirectories.

To return the .c files in a single directory use:

(directory "/Users/foo/c/src/*.c")

Note that DIRECTORY returns a list of pathname objects (not a list of strings).

? (directory
    (make-pathname
      :name "md5"
      :type :wild
      :directory '(:absolute "Lisp" "cl-http" "cl-http-342" "server")))
(#P"/Lisp/cl-http/cl-http-342/server/md5.lisp"
 #P"/Lisp/cl-http/cl-http-342/server/md5.xfasl")

Above uses a pathname object that is created by MAKE-PATHNAME. It returns all the files that match /Lisp/cl-http/cl-http-342/server/md5.* .

This is the same as:

(directory "/Lisp/cl-http/cl-http-342/server/md5.*")

which is shorter, but depends on the Unix pathname syntax.

Huckleberry

Answer 3

The modern Common Lisp library implementing directory listing is IOLIB.

It works like this:

CL-USER> (iolib.os:list-directory "/etc/apt")
(#/p/"trusted.gpg~" #/p/"secring.gpg" #/p/"trustdb.gpg" #/p/"sources.list"
 #/p/"sources.list~" #/p/"apt-file.conf" #/p/"apt.conf.d" #/p/"trusted.gpg"
 #/p/"sources.list.d")

Note that no trailing slash or wildcards are required. It is very robust and can even process file names with incorrectly encoded unicode characters.

Differences compared to CL-FAD:

• The objects you get are IOLIB file paths, a replacement for CL's pathnames which is closer what the underlying OS does.
• IOLIB implements its routines using CFFI, so it works the same on all Lisp implementations (provided IOLIB has a backend for the operating system), in contrast to CL-FAD, which tries to abstract over the implementation's DIRECTORY function with all its quirks.
• In contrast to CL-FAD, iolib deals correctly with symlinks (one major issue with CL-FAD that makes it virtually unusable on platforms other than Windows IMHO).

Longitudinal

Answer 4

I'll add an example that works for me, for the sake of a code snippet. I use osicat (similar to cl-fad) and str.

edit: also with uiop:directory-files. str:contains? could be done with search.

;; searching for "ref".
(setf *data-directory* "~/books/lisp")
(remove-if-not (lambda (it)
                   (str:contains? "ref" (namestring it)))
                (osicat:list-directory *data-directory*))

returns

(#P"~/books/lisp/common-lisp-quick-reference-clqr-a4-booklet-all.pdf"
 #P"~/books/lisp/common-lisp-quick-reference-clqr-a4-consec.pdf"
 #P"~/books/lisp/commonLisp-interactive-approach-reference-buffalo.pdf")

It can certainly be improved my a proper use of wildcards. However that's a snippet you can use right now : )

References:

• https://github.com/CodyReichert/awesome-cl#files-and-directories
• https://lispcookbook.github.io/cl-cookbook/files.html

Hic