Why #' is used before lambda in Common Lisp?

Question

I would like to know why most Common Lisp code I see has things like

(mapcar #'(lambda (x) (* x x)) '(1 2 3))

instead of just

(mapcar (lambda (x) (* x x)) '(1 2 3)),

which seems to work as well. I am beginning to learn Common Lisp, and having some background in Scheme, this intrigues me.

Edit: I know that you need #' with function names because they live in a different namespace than variables. My question is just about #' before lambda, as lambda already returns a function object (I think). The fact that #'-less lambdas work because of a macro expansion just makes it more intriguing...

Promontory

Answer 1

#'foo is an abbreviation for (function foo) by the reader.

In CL, there are several different namespaces, #'foo or (function foo) will return the functional value of foo.

You may want to search for "Lisp-1 vs. Lisp-2", check other Stackoverflow questions, or read an old article by Pitman and Gabriel in order to learn more about the concept of multiple namespaces (also called slots or cells of symbols).

The reason that, in the case of lambda, the #' may be omitted in CL is that it is a macro, which expands thusly (taken from the Hyperspec):

(lambda lambda-list [[declaration* | documentation]] form*)
==  (function (lambda lambda-list [[declaration* | documentation]] form*))
==  #'(lambda lambda-list [[declaration* | documentation]] form*)

#' may still be used for historic reasons (I think that in Maclisp lambdas didn't expand to the function form), or because some people think, that tagging lambdas with sharpquotes may make the code more readable or coherent. There may be some special cases in which this makes a difference, but in general, it doesn't really matter which form you choose.

I guess you can think of it like this: (function (lambda ...)) returns the function (lambda ...) creates. Note that lambda in the CL Hyperspec has both a macro AND a symbol entry. From the latter:

A lambda expression is a list that can be used in place of a function name in certain contexts to denote a function by directly describing its behavior rather than indirectly by referring to the name of an established function.

From the documentation of function:

If name is a lambda expression, then a lexical closure is returned.

I think the difference is also related to calling lambda forms like this: ((lambda ...) ...) where it is treated as a form to be evaluated, vs. (funcall #'(lambda ...) ...). If you want to read more on the topic, there is a c.l.l thread about it.

Some quotes from that thread:

(lambda (x) ... by itself is just some unquoted list structure. It is its appearance as an argument to the FUNCTION special form (function (lambda (x) ... that causes the function object to exist

and:

It's also compounded by the fact that the LAMBDA macro was a rather late addition the ANSI Common Lisp, so all of the really old guys (i.e., like me) learned their lisp when you needed to supply the #' to the lambda expression in the mapping functions. Otherwise the non-existent lambda function would be invoked.

The macro addition changed that, but some of us are too set in our ways to want to change.

Boleyn

I am aware of the namespace distinction. But I expected that since lambda returns a function object directly (or does it?), a call to 'function' or #' would not be needed. Why is it so? – Aponte

Hmmm, so no, the pure lambda does not return a function object... thanks for the explanation. – Aponte

Another link discussing (funcall (lambda ...)) and ((lambda ...)) duality of syntax: letoverlambda.com/textmode.cl/guest/chap4.html#sec_4 – Cachinnate

This is good to know, especially if you're using a relative or precursor to CL. – Virilism

Answer 2

It is best to avoid #' in most instances because it is "mostly" unnecessary and makes your code more verbose. There are a few exceptions when some form of quoting is necessary (see Example 4 below).

Note: All the examples in this post have been tested in Emacs Lisp (GNU Emacs 25.2.1), but they should work identically in any ANSI common lisp. The basic concepts are the same in both dialects.

SIMPLE EXPLANATION
First, let us study a case when it is best to avoid quoting. Functions are first class objects (e.g. treated like any other object, including the ability to pass them to functions and assign them to variables) that evaluate to themselves. Anonymous functions (e.g. lambda forms) are one such example. Try the following on Emacs Lisp (M-x ielm RET) or ANY ANSI common lisp.

((lambda (x) (+ x 10)) 20) -> 30

Now, try the quoted version

(#'(lambda (x) (+ x 10)) 20) -> "function error" or "invalid function..."  

If you use insist on using #', you have to write

(funcall #'(lambda (x) (+ x 10)) 20) -> 30

DETAILED EXPLANATION
To really understand when quoting is required, one must know how Lisp evaluates expressions. Read on. I promise to make this succinct.

You need to know a few basic facts about Lisp:

1. Lisp "always" evaluates every expression. Well, unless the expression is quoted, in which case it is returned unevaluated.
2. Atoms evaluate to themselves. Atomic expressions are NOT lists. Examples include numbers, strings, hash tables, and vectors.
3. Symbols (variable names) store two types of values. They can hold regular values and functional definitions. Hence, Lisp symbols have two slots called cells to store these two types. Non-functional content is typically held in the symbol's value cell and functions in the function cell. The ability to hold both non-functional and functional definitions simultaneously place Emacs Lisp and Common Lisp in the 2-Lisp category. Which of the two cells is used in an expression depends on how the symbol is used -- more specifically its position in a list. In contrast, symbols in some dialects of Lisp, Scheme being the best known, can hold only one value. Scheme has no concept of value and function cells. Such Lisps are collectively called 1-Lisps.

Now, you need a rough understanding of how Lisp evaluates S-expressions (parenthetical expressions). Every S-expression is evaluated roughly as follows:

1. If quoted, return it unevaluated
2. If unquoted, get its CAR (e.g. first element) and evaluate it using the following rules:

a. if an atom, simply return its value (e.g. 3 -> 3, "pablo" -> "pablo")
b. if an S-expression, evaluate it using the same overall procedure
c. if a symbol, return the contents of its function cell

3. Evaluate each of the elements in the CDR of the S-expression (e.g. all but the first element of the list).
4. Apply the function obtained from the CAR to the values obtained from each of the elements in the CDR.

The above procedure implies that any symbol in the CAR of an UNQUOTED S-expression must have a valid functional definition in its function cell.

Now, let's go back to the example from the beginning of the post. Why does

(#'(lambda (x) (+ x 10)) 20)  

generate an error? It does because #'(lambda (x) (+ x 10)), the CAR of the S-expression, is not evaluated by the Lisp interpreter due to the functional quote #'.

#'(lambda (x) (+ x 10))

is not a function, but

(lambda (x) (+ x 10))

is. Keep in mind that the purpose of quoting is to prevent evaluation. On the other hand, a lambda form evaluates to itself, a functional form, which is valid as the CAR of an UNQUOTED list. When Lisp evaluates the CAR of

((lambda (x) (+ x 10)) 20)  

it gets (lambda (x) (+ x 20)), which is a function that can be applied to the rest of the arguments in a list (provided the length of the CDR equals the number of arguments allowed by the lambda expression). Hence,

((lambda (x) (+ x 10)) 20) -> 30  

The question is thus when to quote functions or symbols holding functional definitions. The answer is almost NEVER unless you do things "incorrectly." By "incorrectly" I mean that you place a functional definition in a symbol's value cell or functional cell when you should do the opposite. See the following examples to gain a better understanding:

EXAMPLE 1 - Functions Stored in Value Cells
Suppose you need to use apply with a function that expects a variable number of arguments. One such example is the symbol +. Lisp treats + as a regular symbol. The functional definition is stored in +'s functional cell. You can assign a value to its value cell if you like using

(setq + "I am the plus function").  

If you evaluate

+ -> "I am the plus function"

However, (+ 1 2) still works as expected.

(+ 1 2) -> 3

The function apply is quite useful in recursion. Suppose you want to sum all the elements in a list. You CANNOT write

(+ '(1 2 3)) -> Wrong type...  

The reason is that + expects its arguments to be numbers. apply solves this problem

(apply #'+ '(1 2 3)) -> (+ 1 2 3) -> 6  

Why did I quote + above? Remember the evaluation rules I outlined above. Lisp evaluates the symbol apply by retrieving the value stored in its function cell. it gets a functional procedure it can apply to a list of arguments. However, if I do not quote +, Lisp will retrieve the value stored in its value cell because it is NOT the first element in the S-expression. Because we set +'s value cell to "I am the plus function," Lisp does not get the functional definition held in +'s function cell. Actually, if we had not set its value cell to "I am the plus function," Lisp would have retrieved nil, which is NOT a function, as required by apply.

Is there a way to use + unquoted with apply? Yes, there is. You can just evaluate the following code:

(setq + (symbol-function '+))  
(apply + '(1 2 3))  

This will evaluate to 6, as expected because as Lisp evaluates (apply + '(1 2 3)) it now finds the functional definition of + stored in +'s value cell.

EXAMPLE 2 - Storing Functional Definitions in Value Cells
Suppose you store a functional definition in the symbol's value cell. This is achieve as follows:

(setq AFunc (lambda (x) (* 10 x)))

Evaluating

(AFunc 2)

generates an error because Lisp cannot find a function in AFunc's function cell. You get around this by using funcall, which tells Lisp to use the value in the symbol's value cell as a functional definition. You do this using "funcall."

(funcall AFunc 2)

Assuming the functional definition stored in the symbol's value cell is valid,

(funcall AFunc 2) -> 20  

You could avoid having using funcall by placing the lambda form in the symbol's function cell using fset:

(fset 'AFunc (lambda (x) (* 10 x)))  
(AFunc 2)  

This code block will return 20 because lisp finds a functional definition in AFunc's function cell.

EXAMPLE 3 - Local Functions
Say you are writing a function and need a function that won't be used anywhere else. A typical solution is to define a function valid only inside the scope of the main one. Try this:

(defun SquareNumberList (AListOfIntegers)
    "A silly function with an unnecessary
   local function."
  (let ((Square (lambda (ANumber) (* ANumber ANumber))))
    (mapcar Square AListOfIntegers)
    )
  )  

(SquareNumberList '(1 2 3))  

This code block will return

(1 4 9)  

The reason Square is not quoted in the above example is that S-expression is evaluated according to the rules I outlined above. First, Lisp pulls the functional definition of mapcar. Next, Lisp pulls the contents of its second argument's (e.g. 'Square) value cell. Finally, it returns (1 2 3) unevaluated for the third argument.

EXAMPLE 4 - Contents of a Symbol's Value and Function Cells
Here is one case when quotes are required.

(setq ASymbol "Symbol's Value")  
(fset 'ASymbol (lambda () "Symbol's Function"))  
(progn  
  (print (format "Symbol's value -> %s" (symbol-value 'ASymbol)))  
  (print (format "Symbol's function -> %s" (symbol-function 'ASymbol)))
  )    

The above code will evaluate to

"Symbol's value -> Symbol's Value"  
"Symbol's function -> (lambda nil Symbol's Function)"  
nil

Quote are required in

(fset 'ASymbol (lambda () "Symbol's Function"))  

and

(symbol-value 'ASymbol)  

and

(symbol-function 'ASymbol)  

because otherwise Lisp would get the value of ASymbol in each case, preventing fset, symbol-value, and symbol-function from working correctly.

I hope this lengthy post proves useful to someone.

Rope

(setf AFunc (lambda (x) (* 10 x))) followed by (AFunc 2) produces an error: The function COMMON-LISP-USER::AFUNC is undefined. – Gawain

You are correct. I had a typo in (setf AFunc (lambda (x) (* 10 x))) followed by (AFunc 2). I should have typed (fset 'AFunc (lambda (x) (* 10 x))) which returns (closure (t) (x) (* 10 x)) in Emacs Lisp. Evaluating (AFunc 2) returns 20. fset sets the function cell of a symbol. I have corrected the text accordingly. – Rope