Common Lisp Double-Backquote, Unquote, Quote, Unquote sequence?

Question

I'm reading Let Over Lambda, which deals with some pretty deeply layered macro authoring. It's fascinating and I'm mostly managing to keep up with it.

In Chapter 4 Hoyte implements reader macros for CL-PPCRE match and replace functions, such that you can do things like:

(#~m/(foo|bar)\d+/ "Some foo99")    ; matches!
(#~s/foo(\d+)/bar\1/, "Some foo99") ; "Some bar99

In order to achieve this, we define a macro that uses the double-backquote, since it is actually expanded by a wrapper macro, which needs the quoted value (it returns a lambda form). Within the quasi-quoted list, there is some use of the following sequence ,',varname, which I can't get my head around. What does the initial ,' do here?

(defmacro! pcre/match-lambda-form (o!args)
  "Expands to a lambda that applies CL-PPCRE:SCAN"
  ``(lambda (,',g!str)
      (cl-ppcre:scan ,(car ,g!args)
                     ,',g!str)))

Actually, it's probably better that I distill that down to something that uses just defmacro, for clarity if you haven't read the book. str is a symbol and args is a list:

(defmacro pcre/match-lambda-form (args)
  "Expands to a lambda that applies CL-PPCRE:SCAN"
  ``(lambda (,',str)
      (cl-ppcre:scan ,(car ,args)
                     ,',str)))

Are the quotes basically double-quoting the inner parts, so that the result can be unquoted twice? Effectively putting 'str into the expanded form, instead of just str?

EDIT | Thanks to Terje D. and some playing around in the REPL, this is pretty much the situation:

(defvar a 42)

(equal ``(,,a)  '(list 42)) ; T
(equal ``(,a)   '(list a))  ; T
(equal ``(,',a) ''(42))     ; T
(equal ``(a)    ''(a))      ; T (obviously)

So:

• Doubly-unquoted, form is fully expanded.
• Singly-unquoted, form is not expanded.
• Unquoted with a comma, form is fully expanded and the result quoted.

Phylactery

your last defmacro references str. What is it? Where is it defined? Can you show how pcre/match-lambda-form is called? – Filum

Argh, this is going to take some explaining as there are many layers of macro going on in reality. This book is pushing boundaries with macro authoring (for fun, and for education). str is a symbol that has been defined by an enclosing LET and represents a variable name that is safe to use. – Phylactery

pcre/match-lambda-form is called like so: (pcre/match-lambda-form '("foo")), where the "foo" has been read by a stream reader processing strings of the form #~m/foo/. – Phylactery

I'm reading it in print form, but it's (partially) available online too: letoverlambda.com/index.cl/toc – Phylactery

letoverlambda.com/index.cl/guest/chap4.html#sec_4 You'll also need the definition of defmacro! from chapter 3, which uses defmacro/g!, which uses... (you get the idea) :) – Phylactery

Working code available here: github.com/d11wtq/lol-notes/blob/master/pcre.lisp – Phylactery

Great question! Have you read 'On Lisp' by Paul Graham? In chapter 16 'Macro-Defining Macros' he briefly explains how to construct nested backquotes. Unfortunately he doesn't spend a lot of time on the subject... – Daggna

I haven't, but it's next on my list, just because this book constantly refers to it (with admiration). Hoyte wrote this because he felt Graham didn't go into enough detail on macros, while he's of the opinion they provide such great power, they should be thoroughly understood by serious lisp programmers. – Phylactery

@Daggna thanks for the link by the way. I didn't realize it was online :) – Phylactery

Mmm... (equal ``(,,a) '(list 42)) gives NIL to me when tried in the SBCL REPL. I too would like to understand this kind of macros. I've still not started with them, though. – Moltke

Hmm, yeah SBCL is doing something different. I'm not sure who's right or wrong (I was using CLISP). – Phylactery

There is no right or wrong here. According to the Hyperspec, section 2.4.6 (Backquote): An implementation is free to interpret a backquoted form F1 as any form F2 that, when evaluated, will produce a result that is the same under equal as the result implied by the above definition, provided that the side-effect behavior of the substitute form F2 is also consistent with the description given above. I.e the forms must be fully evaluated before any comparison is done. – Downhill

Answer 1

During evaluation of a doubly backquoted form, the inner backquote is handled first, and the result is a singly backquoted form. During evaluation of the inner backquoted form, only elements preceeded by two commas are evaluated. However, the result of evaluating these doubly unquoted elements are still (singly) unquoted, and are thus evaluated again when the resulting singly backquoted form are evaluated. To achieve evaluation only in the inner backquoted form, an ordinary quote has to be inserted, resulting in ,',.

See how

(let ((tmp (gensym)))
    ``(lambda (,tmp ,,tmp ,',tmp) ()))

evaluates to

`(LAMBDA (,TMP ,#:G42 #:G42) nil)

Downhill

Wow, that's a bit complex. I'm just playing around with some basic double-backquoting and those variations of unquoting now, to see what happens in each case. I get it now, thanks :) – Phylactery

@Phylactery You may also want to have a look at appendix C (backquote) of 'Common Lisp the Language' available at cs.cmu.edu/Groups/AI/html/cltl/clm/node1.html , which contains a sample implementation and examples of backquote syntax. – Downhill

Answer 2

The ,',X trick is used to shield the X from another evaluation.

See how:

     (setq a 'fn)
     (let ((x 'a)) ``(,,x ,',x)) ==>  `(,a a) ==> (fn a)

     ;; ``,',X ==> `,(quote "the value of X") ==> "the value of X"

     ;; ``,,X  ==> `,"the value of X" ==> "the value of the value of X"

Pentameter