Lisp unit tests for macros conventions and best practices

Question

I find it hard to reason about macro-expansion and was wondering what the best practices were for testing them.

So if I have a macro, I can perform one level of macro expansion via macroexpand-1.

(defmacro incf-twice (n)
  `(progn
     (incf ,n)
     (incf ,n)))

for example

(macroexpand-1 '(incf-twice n))

evaluates to

(PROGN (INCF N) (INCF N))

It seems simple enough to turn this into a test for the macro.

(equalp (macroexpand-1 '(incf-twice n))
  '(progn (incf n) (incf n)))

Is there an established convention for organizing tests for macros? Also, is there a library for summarizing differences between s-expressions?

Logotype

I would test the end effect of the macro, not the intermediate expansion. Great question though, I'll look forward to the answers. – Cheep

Answer 1

Generally testing macros is not one of the strong parts of Lisp and Common Lisp. Common Lisp (and Lisp dialects in general) uses procedural macros. The macros can depend on the runtime context, the compile-time context, the implementation and more. They also can have side effects (like registering things in the compile-time environment, registering things in the development environment and more).

So one might want to test:

• that the right code gets generated
• that the generated code actually does the right thing
• that the generated code actually works in code contexts
• that the macro arguments are actually parsed correctly in case of complex macros. Think loop, defstruct, ... macros.
• that the macro detects wrongly formed argument code. Again, think of macros like loop and defstruct.
• the side effects

From above list on can infer that it is best to minimize all these problem areas when developing a macro. BUT: there are really really complex macros out there. Really scary ones. Especially those who are used to implemented new domain specific languages.

Using something like equalp to compare code works only for relatively simple macros. Macros often introduce new, uninterned and unique symbols. Thus equalp will fail to work with those.

Example: (rotatef a b) looks simple, but the expansion is actually complicated:

CL-USER 28 > (pprint (macroexpand-1 '(rotatef a b)))

(PROGN
  (LET* ()
    (LET ((#:|Store-Var-1234| A))
      (LET* ()
        (LET ((#:|Store-Var-1233| B))
          (PROGN
            (SETQ A #:|Store-Var-1233|)
            (SETQ B #:|Store-Var-1234|))))))
  NIL)

#:|Store-Var-1233| is a symbol, which is uninterned and newly created by the macro.

Another simple macro form with a complex expansion would be (defstruct s b).

Thus one would need a s-expression pattern matcher to compare the expansions. There are a few available and they would be useful here. One needs to make sure in the test patterns that the generated symbols are identical, where needed.

There are also s-expression diff tools. For example diff-sexp.

Smudge

Answer 2

I agree with Rainer Joswig's answer; in general, this is a very difficult task to solve because macros can do a whole lot. However, I would point out that in many cases, the easiest way to unit test your macros is by making the macros do as little as possible. In many cases, the easiest implementation of a macro is just syntactic sugar around a simpler function. E.g., there's a typical pattern of with-… macros in Common Lisp (e.g., with-open-file), where the macro simply encapsulates some boilerplate code:

(defun make-frob (frob-args)
  ;; do something and return the resulting frob
  (list 'frob frob-args))

(defun cleanup-frob (frob)
  (declare (ignore frob))
  ;; release the resources associated with the frob
  )

(defun call-with-frob (frob-args function)
  (let ((frob (apply 'make-frob frob-args)))
    (unwind-protect (funcall function frob)
      (cleanup-frob frob))))

(defmacro with-frob ((var &rest frob-args) &body body)
  `(call-with-frob
    (list ,@frob-args)
    (lambda (,var)
      ,@body)))

The first two functions here, make-frob and cleanup-frob are relatively straightforward to unit test. The call-with-frob is a bit harder. The idea is that it's supposed to handle the boilerplate code of creating the frob and ensuring that the cleanup call happens. That's a bit harder to check, but if the boilerplate only depends on some well defined interfaces, then you'll probably be able to create mock up a frob that can detect whether it's cleaned up correctly. Finally, the with-frob macro is so simple that you can probably test it the way you've been considering, i.e., checking its expansion. Or you might say that it's simple enough that you don't need to test it.

On the other hand, if you're looking at a much more complex macro, such as loop, which is really a kind of compiler in its own right, you're almost certainly already going to have the expansion logic in some separate functions. E.g., you might have

(defmacro loop (&body body)
  (compile-loop body))

in which case you really don't need to test loop, you need to test compile-loop, and then you're back in the realm of your usual unit testing.

Sheepshead

Answer 3

I'd generally just test the functionality, not the shape of the expansion.

Yes, there are all kinds of contexts and surroundings that might influence what happens, but if you rely on such things, it should be no problem to set them up the same for your test.

Some common cases:

• binding macros: test that the variables are bound as intended inside and that any shadowed outside variables are unaffected
• unwind-protect wrappers: provoke a nonlocal exit from the inside and check that the cleanup is working
• definition/registration: test that you can define/register what you want and use it afterwards

Dalton