Why is it possible to use unquote-splicing on a non-list at the end of a quasiquoted list?

Question

The quasiquoted list `(1 ,@2 3) is invalid because 2 is not a list. However, `(1 2 ,@3) is valid and will return a dotted list: (1 2 . 3). I observe this result in Common Lisp and Scheme. Why is it possible to use unquote-splicing for non-lists at the end of a quasiquoted list? Why is the result a dotted list?

Pasho

Answer 1

The expression `(1 2 ,@3) is not valid in either Scheme or Common Lisp.

---

Scheme

In R6RS Scheme (and similarly for R5RS), the behavior is not specified for operating on a non-list with unquote-splicing. The R6RS Scheme Standard requires (11.17 Quasiquotation):

If an (unquote-splicing <expression> ...) form appears inside a <qq template>, then the <expression>s must evaluate to lists....

---

Common Lisp

The Common Lisp HyperSpec says first that (2.4.6 Backquote):

If a comma is immediately followed by an at-sign, then the form following the at-sign is evaluated to produce a list of objects. These objects are then "spliced" into place in the template.

In the sub-expression ,@3, 3 does not evaluate to a list. This seems a pretty strong argument that the expression is not valid. Even if the 3 were magically placed in a list before splicing, this would not result in a dotted list. The HyperSpec goes on to provide a formal summary of backquote syntax. The part of interest is:

• `(x1 x2 x3 ... xn . atom) may be interpreted to mean

  (append [ x1] [ x2] [ x3] ... [ xn] (quote atom))

  where the brackets are used to indicate a transformation of an xj as follows:

  -- [form] is interpreted as (list `form), which contains a backquoted form that must then be further interpreted.

  -- [,form] is interpreted as (list form).

  -- [,@form] is interpreted as form.

So in Common Lisp the original expression, which is equivalent to `(1 2 ,@3 . nil), can be interpreted as:

(append (list `1) (list `2) 3 (quote nil))

But, this is not a valid call to append, which requires proper lists for all arguments except the last one. So, there seems to be no support for the idea that the original expression was valid.

The fact that it worked for the OP in both Scheme and Common Lisp probably comes down to similar definitions for the backquote macro across different implementations. These definitions all seem to expect that the form following ,@ will evaluate to a list; when that is not the case, the observed behavior (production of a dotted list) can't be relied upon, according to the standard. That said, I tested Chez Scheme, Guile Scheme, MIT Scheme, SBCL, CCL, and CLisp: all of them exhibited the same behavior reported by OP.

---

An Interesting Case

I also tested against an implementation of the backquote macro by Guy Steele and published in CLTL2. This case is more interesting. This backquote implementation in CLTL2 is meant for exploring the behavior of backquote expressions and has an optional code simplification phase. Here $ corresponds to a backquote, and %@ corresponds to ,@. Without code simplification, the result of expanding the original expression is:

CL-USER> (setf *bq-simplify* nil)
NIL
CL-USER> (try '("$(1 2 %@3)"))
`(1 2 ,@3) = (APPEND (LIST '1) (LIST '2) 3 'NIL)

This corresponds to the expression which was expected above from reading the description in the HyperSpec. But note that this expression will not compile:

CL-USER> (append (list 1) (list 2) 3 nil)

The value
  3
is not of type
  LIST
[Condition of type TYPE-ERROR]

But, when code simplification is turned on:

CL-USER> (setf *bq-simplify* t)
T
CL-USER> (try '("$(1 2 %@3)"))
`(1 2 ,@3) = (LIST* '1 '2 3)

This "simplified" expression is valid, and evaluates to a dotted list:

CL-USER> (list* 1 2 3)
(1 2 . 3)

My conclusion is that the expression following ,@ must be a list per the Common Lisp standard, but some common implementations either do some form of code simplification similar to what is shown in CLTL2 or otherwise expand the backquote form in such a way that it appears that a non-list form can follow ,@. Don't rely on this, as it is difficult to say when it won't work.

Tuberculin

Interestingly, `(1 2 ,@3) is valid in Racket. The documentation gives an example: (quasiquote (0 (unquote-splicing 1))) returns '(0 . 1). "... if the cdr of the relevant quoted pair is empty, then expr need not produce a list, and its result is used directly in place of the quoted pair (in the same way that append accepts a non-list final argument)." – Pasho

@Pasho -- that is interesting; it would be interesting to hear from someone involved in the implementation of Racket about the rationale for making this explicit. – Tuberculin

Answer 2

Proper lists

Other answers detail this part already, let me rephrase it quickly: lists are either proper lists or improper lists; improper lists are either ciruclar lists or dotted lists.

In particular, a dotted-list is a non-circular list whose last cons cell has a non-list cdr slot.

Some functions are expected to work only when given proper lists, other have no such restrictions, typically because checking for this property is not without costs. Writing (cons 1 2) creates such a list. Now, if the splicing syntax expands as (cons x y) where y is a non-list, you'll have a dotted-list too.

Slicing

Chapter 2.4.6 Backquote states (emphasis mine):

If a comma is immediately followed by an at-sign, then the form following the at-sign is evaluated to produce a list of objects.

I see no other note indicating that splicing a non-list value might be part of a conforming program, even if it results in practice to an improper list when splicing occurs at the end of the list.

The section also states that:

`((,a b) ,c ,@d)

will be interpreted as if it were

(append (list (append (list a) (list 'b) 'nil)) (list c) d 'nil)

but it could also be legitimately interpreted to mean any of the following:

(append (list (append (list a) (list 'b))) (list c) d)
(append (list (append (list a) '(b))) (list c) d)
(list* (cons a '(b)) c d)
(list* (cons a (list 'b)) c d)
(append (list (cons a '(b))) (list c) d)
(list* (cons a '(b)) c (copy-list d))

Expanding as calls to standard functions like list* or append make that corner case naturally produce improper lists, but notice that the first and last example do not allow splicing for the last element. For instance:

(list* (cons a '(b)) c (copy-list d))

The above form complains when d is not a list, because COPY-LIST only works with lists (either proper or dotted, but here it would be given a non-list value, e.g. a number).

It is thus my understanding that this expression:

`(1 2 ,@3)

is in fact invalid in Common Lisp and happens to works only by accident. Writing it could expose you to portability problems.

Conclusion

Why is it possible to use unquote-splicing for non-lists at the end of a quasiquoted list?

It happens by chance in your implementation because backquotes, unquotes and splicing are rewritten/expanded as list building functions that may produce dotted lists in that corner case, without signaling an error.

Why is the result a dotted list?

Because the code probably is interpreted as a call to append where the last item is a non-list value, which results in a dotted list. It does not work when splicing in the middle of a form, because append expects proper lists for all but the last arguments.

Bonus

Today I learned about ,. which is like ,@ except is may use nconc instead of append

Slaveholder

Answer 3

To construct a list of 3 variables with quasiquoting, one could write

`(,x ,y ,z)

This can be desugared to

(cons x (cons y (cons z nil)))

What if z is a list, and we want to splice its contents into the resulting list? Then instead of creating a new cons for it, we simply put it at the tail. When we write

`(,x ,y ,@z)

this desugars to

(cons x (cons y z))

If z happens to not actually be a list, then the result is a legal value, although not a proper list. For example, (1 2 . 3), as you observed. You will see the same result if you write the desugared form explicitly, as (cons 1 (cons 2 3)). If this also troubles you, you may want to investigate the concept of improper lists in general, or dotted pair notation.

Subvert

Answer 4

'(1 2 3) is actually (cons 1 (cons 2 (cons 3 nil))).

So, the last element in '(1 2 3) is not a 'non-list', but a (cons 3 nil) i.e. (list 3), and thus can be spliced.

Noctule

I don't really agree with this. If the 3 in the quasiquoted list is secretly (cons 3 nil), then why isn't the 2 secretly (cons 2 (cons 3 nil))? It should be legal to splice at any position, if your argument holds. – Subvert

Well it's just how the backquote macro operates I believe :/ – Noctule