Why is it necessary to recompile the definitions that use a Common Lisp macro when we modify it?

Question

This is one doubt that always made me curious.

Why is it necessary to also recompile the definitions that use a Common Lisp macro when we modify it (besides, obviously, recompile the modified Common Lisp macro)?

Thanks in advance.

Charteris

Answer 1

Common Lisp macros are code substitutions, they happen at "macro expansion time". For interpreted code, this may be "run time". For compiled code, it's almost always during "compile time".

Once the code has been substituted, there is no reference to the macro in the resulting compiled code (the original source on disk still has this, if the macro usage has been saved to a file).

While there would be some convenience in a lisp system that saved all "pre-macro expansion" function bodies and kept track of what macros had been used where and automatically re-compile anything that uses a given macro (possibly recursively), that was left out of the standard.

Typically, I write my utility macros quite early in development, so there's not much need, for me, to have this functionality and I am on the whole happier without it than I would be with it (it cuts down on the running image size quite a bit, not needing to track all of that).

Clarisaclarise

Answer 2

Note that it may be necessary.

The reason is that macros in compiled implementations are expanded only once at compile time and the macro itself is not present in the generated code.

To be more clear consider

(defmacro badsquare (x)
  `(* ,x ,x))

(defun f (x)
  (badsquare (+ x 3))

when the compiler analyzes f code will expand it to

(defun f (x)
  (* (+ x 3) (+ x 3)))

and the references to the badsquare macro are not necessarily present any more so if you redefine the badsquare macro to something else this will have no effect on f unless you also recompile it.

Flaunt

Answer 3

Recompiling the uses of a macro which has changed are only necessary when:

• The old macro had a bug. What it means for a macro to have bug is that it generates bad code. The bad code has to be regenerated by the fixed macro.

• The old macro generated sucky code. We want to recompile the code with the latest and greatest macro that generates the nicest code.

• The run-time support for the old macro is going away. The old macro generated code which makes calls to special run-time support functions which are also provided by the package. Some of those functions are going away in the new version of the macro package, or are changing in backward-incompatible ways. So when the new functions are installed, existing compiled code will not work correctly. Corrolary: When maintaining macro packages that have run-time support, keep in mind code that might not be recompiled and have a plan for maintaining compatibility and imposing obsolescence.

• Consistency. You've modified multiple macros several times, but you have recompiled only some functions or source files depending on the macro, while neglecting others. You're thus testing a "Frankenstein's monster" image which contains macro expansions that cannot be rebuilt any more from the source code. And any testing you are doing is only valid for that image. When things break, it could just be because of the inconsistency, and so you waste time chasing nonexistent bugs. Or vice versa: something works right, but only due to the combination of stale and fresh code. You want to rebuild everything from scratch so that you have a well-defined baseline for testing the software or packaging a release.

• Interface change. The macro's syntax has changed, and so code has to be updated to use the new macro. While it is possible that the old binary code which was expanded with the old macro can continue to be used, you then have the situation of running binaries that are stale with respect to the updated source code (bringing us back to the previous point of consistency).

If none of these conditions apply, then we can allow ourselves to be lazy in recompiling code that depends on macros.

Procora