Why the function/macro dichotomy?

Question

Why is the function/macro dichotomy present in Common Lisp?

What are the logical problems in allowing the same name representing both a macro (taking precedence when found in function position in compile/eval) and a function (usable for example with mapcar)?

For example having second defined both as a macro and as a function would allow to use

(setf (second x) 42)

and

(mapcar #'second L)

without having to create any setf trickery.

Of course it's clear that macros can do more than functions and so the analogy cannot be complete (and I don't think of course that every macro shold also be a function) but why forbidding it by making both sharing a single namespace when it could be potentially useful?

I hope I'm not offending anyone, but I don't really find a "Why doing that?" response really pertinent... I'm looking for why this is a bad idea. Imposing an arbitrary limitation because no good use is known is IMO somewhat arrogant (sort of assumes perfect foresight).

Or are there practical problems in allowing it?

Superstratum

Well, if you want to keep your cake and eat it too... Use compiler macros. Look for define-compiler-macro – Poultice

can you give a use case where this would be useful? – Grist

@Dirk. Yeah... that is also something curious, why are compiler macros really needed if you have macros? Just regular macros (with may be the option of being expanded every time during debugging) would seem sufficient. – Superstratum

Issue DEFINE-COMPILER-MACRO explains the use cases for compiler macros. – Yore

@gareth: Indeed there is a passage in that document that is very pertinent to my question ... "DEFMACRO is not usable for this purpose since it requires use of the symbol-function cell, which would prevent the functional definition from being active in the compilation environment.". Basically says that you may need compiler macros for efficiency because macros cannot be also functions (the point I am questioning). – Superstratum

It helps to understand the constraints of the standardization process. It would have been no good to make an elegant standard if the Lisp vendors didn't agree to implement it. So many compromises were necessary in order to make a successful standard with widespread adoption. – Yore

@Gareth Rees: I agree 100%. I simply was not sure this dichotomy was an historical/political accident or a deliberate logical choice because of implications I wasn't able to see. – Superstratum

Answer 1

I think that Common Lisp's two namespaces (functions and values), rather than three (macros, functions, and values), is a historical contingency.

Early Lisps (in the 1960s) represented functions and values in different ways: values as bindings on the runtime stack, and functions as properties attached to symbols in the symbol table. This difference in implementation led to the specification of two namespaces when Common Lisp was standardized in the 1980s. See Richard Gabriel's paper Technical Issues of Separation in Function Cells and Value Cells for an explanation of this decision.

Macros (and their ancestors, FEXPRs, functions which do not evaluate their arguments) were stored in many Lisp implementations in the symbol table, in the same way as functions. It would have been inconvenient for these implementations if a third namespace (for macros) had been specified, and would have caused backwards-compatibility problems for many programs.

See Kent Pitman's paper Special Forms in Lisp for more about the history of FEXPRs, macros and other special forms.

(Note: Kent Pitman's website is not working for me, so I've linked to the papers via archive.org.)

Yore

Thanks for the link (worked for me). I never heard before about those macro precursors... and actually I always wondered about "special" operators too and why they're allegedly needed. – Superstratum

Answer 2

Macros and Functions are two very different things:

• macros are using source (!!!) code and are generating new source (!!!) code

• functions are parameterized blocks of code.

Now we can look at this from several angles, for example:

a) how do we design a language where functions and macros are clearly identifiable and are looking different in our source code, so we (the human) can easily see what is what?

or

b) how do we blend macros and functions in a way that the result is most useful and has the most useful rules controlling its behavior? For the user it should not make a difference to use a macro or a function.

We really need to convince ourselves that b) is the way to go and we would like to use a language where macros and functions usage looks the same and is working according to similar principles. Take ships and cars. They look different, their use case is mostly different, they transport people - should we now make sure that the traffic rules for them are mostly identical, should we make them different or should we design the rules for their special usage?

For functions we have problems like: defining a function, scope of functions, life-time of functions, passing functions around, returning functions, calling functions, shadowing of functions, extension of functions, removing the definition a function, compilation and interpretation of functions, ...

If we would make macros appear mostly similar to functions, we need to address most or all above issues for them.

In your example you mention a SETF form. SETF is a macro that analyses the enclosed form at macro expansion time and generates code for a setter. It has little to do with SECOND being a macro or not. Having SECOND being a macro would not help at all in this situation.

So, what is a problem example?

(defmacro foo (a b)
  (if (and (numberp b) (zerop b))
      a
    `(- ,a ,b)))

(defun bar (x list)
  (mapcar #'foo (list x x x x) '(1 2 3 4)))

Now what should that do? Intuitively it looks easy: map FOO over the lists. But it isn't. When Common Lisp was designed, I would guess, it was not clear what that should do and how it should work. If FOO is a function, then it was clear: Common Lisp took the ideas from Scheme behind lexically scoped first-class functions and integrated it into the language.

But first-class macros? After the design of Common Lisp a bunch of research went into this problem and investigated it. But at the time of Common Lisp's design, there was no wide-spread use of first-class macros and no experience with design approaches. Common Lisp is standardizing on what was known at the time and what the language users thought necessary to develop (the object-system CLOS is kind of novel, based on earlier experience with similar object-systems) software with. Common Lisp was not designed to have the theoretically most pleasing Lisp dialect - it was designed to have a powerful Lisp which allows the efficient implementation of software.

We could work around this and say, passing macros is not possible. The developer would have to provide a function under the same name, which we pass around.

But then (funcall #'foo 1 2) and (foo 1 2) would invoke different machineries? In the first case the function fooand in the second case we use the macro foo to generate code for us? Really? Do we (as human programmers) want this? I think not - it looks like it makes programming much more complicated.

From a pragmatic point of view: Macros and the mechanism behind it are already complicated enough that most programmers have difficulties dealing with it in real code. They make debugging and code understanding much harder for a human. On the surface a macro makes code easier to read, but the price is the need to understand the code expansion process and result. Finding a way to further integrate macros into the language design is not an easy task.

readscheme.org has some pointers to Macro-related research wrt. Scheme: Macros

What about Common Lisp

Common Lisp provides functions which can be first-class (stored, passed around, ...) and lexically scoped naming for them (DEFUN, FLET, LABELS, FUNCTION, LAMBDA).

Common Lisp provides global macros (DEFMACRO) and local macros (MACROLET).

Common Lisp provides global compiler macros (DEFINE-COMPILER-MACRO).

With compiler macros it is possible to have a function or macro for a symbol AND a compiler macro. The Lisp system can decide to prefer the compiler macro over the macro or function. It can also ignore them entirely. This mechanism is mostly used for the user to program specific optimizations. Thus it does not solve any macro related problems, but provides a pragmatic way to program global optimizations.

Yellowhammer

Just to clarify about second second. If second is defined with a macro like (defmacro second (x) (list 'cadr x)) then there is no need for (defun (setf second) ...) to have it working because setf already expands macros, but becomes impossible to use it with mapcar and you've to do tricks like (mapcar (lambda (x) (second x)) ...) because (mapcar #'second ...) would not work. About having (funcall #'foo 1 2) being different from (foo 1 2) it means that they COULD be different if YOU wanted them to be different. What's the bad part of it? – Superstratum

@6502: but that does not help at all in the general case. Now you need to define a place (!) via a macro, instead of registering it with a SETF mechanism. Why should a macro expansion be a place? It could be or it could not be. You would need to document it now which is expanding into a place and which is not. What if I want the form to be a place and have a different macro expansion? – Yellowhammer

@6502: (funcall #'foo 1 2) and (foo 1 2) do the same for functions. It is only a different way to write the same thing. Why should I make it different? I could also define (foo 1 2) to first look for a macro and use that if it exists. It works if the form exists in the source. If FOO gets passed around as a first class function and gets called, only the function can be used - not the macro. You propose to add more complication to the language, making debugging harder. The CL standard provides that in the global case with compiler macros, with the benefit of optimization - not widely used btw.. – Yellowhammer

sorry but I fail to see why THE POSSIBILITY of having a function and a macro with the same name would make the language more complex. Just don't use that if for you is confused. About second being automatically setf-able because of being a macro is once again just a POSSIBILITY... nothing would prevent you to define a custom setf-handler for that (in CL setf CAN expand a macro but "only after exhausting all other possibilities other than expanding into a call to a function named (setf reader)"). – Superstratum

@6502: features which only are a 'possibility' are useless anyway. It is already difficult to determine what a form does. Now you propose to add even more complication to it, without a good idea how it should integrate into the language. It is also not only my problem to use it, but there are people who provide libraries possibly using it. – Yellowhammer

@6502: SETF can expand a macro. If I want to have a SETF expander for SECOND, I'd write one. Why should I write it as a macro? When I write a macro for SECOND, why should it look like a place? What you propose works implicitly. It is much better if our intentions are visible in the source code. If you want to define a new SETF expander, use the provide forms. If you want a macro, define it. Don't mix everything. It is much better to have self-documenting forms, instead of forms where I have to document what they should do and what kind of side-effects the use or provide. – Yellowhammer

Just to recap do you think that this was the point because macros and functions have the same namespace (not like variables, structures, tags, compiler macros etc... where each one has its own separate namespace)? Can you point to a document where this discussion is described? Until then I think the best explanation for why there is a function/macro dichotomy is historical. I looked a bit in the documents you already pointed to but didn't find relevant instances (mostly are about forced hygene obsession). – Superstratum

@6502: when Lisp was invented, there were no macros in Lisp. There were only functions, no macros. Lisp was developed as language around functions. When Macros were added the theoretical problems were not clear - not even those of the Functional part of Lisp. Macros later replaced Fexprs - macros were compilable, Fexprs not. Some Lisp dialects had Fexprs for backwards compatibility, but it was phased out then. It was not thought that Macros would supersede functions or should. By default Lisp is a functional language, not a macro programming language. – Yellowhammer

Answer 3

Because then the exact same name would represent two different objects, depending on the context. It makes the programme unnecessarily difficult to understand.

Chaffee

That can't be the whole story, though, because your argument would seem to apply to functions and values, and there are different namespaces for those two things. – Yore

With macros, you can introduce new syntax (you control, which arguments must be evaluated, and which not). With functions, the language determins the behaviour – Poultice

I don't mean that EVERY macro should also be a function, but I wonder why I cannot have both a macro and a function (usable with funcall) with the same name. – Superstratum

In Common Lisp, the symbol list can be a variable/parameter in a function that also calls (list ...), so that (list list) makes sense. The argument against multiple namespaces disambiguated by syntax/context is a ship that sailed long ago. – Humphries

Answer 4

My TXR Lisp dialect allows a symbol to be simultaneously a macro and function. Moreover, certain special operators are also backed by functions.

I put a bit of thought into the design, and haven't run into any problems. It works very well and is conceptually clean.

Common Lisp is the way it is for historic reasons.

Here is a brief rundown of the system:

• When a global macro is defined for symbol X with defmacro, the symbol X does not become fboundp. Rather, what becomes fboundp is the compound function name (macro X).

• The name (macro X) is then known to symbol-function, trace and in other situations. (symbol-function '(macro X)) retrieves the two-argument expander function which takes the form and an environment.

• It's possible to write a macro using (defun (macro X) (form env) ...).

• There are no compiler macros; regular macros do the job of compiler macros.

• A regular macro can return the unexpanded form to indicate that it's declining to expand. If a lexical macrolet declines to expand, the opportunity goes to a more lexically outer macrolet, and so on up to the global defmacro. If the global defmacro declines to expand, the form is considered expanded, and thus is necessarily either a function call or special form.

• If we have both a function and macro called X, we can call the function definition using (call (fun X) ...) or (call 'X ...), or else using the Lisp-1-style dwim evaluator (dwim X ...) that is almost always used through its [] syntactic sugar as [X ...].

• For a sort of completeness, the functions mboundp, mmakunbound and symbol-macro are provided, which are macro analogs of fboundp, fmakunbound and symbol-function.

• The special operators or, and, if and some others have function definitions also, so that code like [mapcar or '(nil 2 t) '(1 0 3)] -> (1 2 t) is possible.

Example: apply constant folding to sqrt:

1> (sqrt 4.0)
2.0
2> (defmacro sqrt (x :env e :form f)
     (if (constantp x e)
       (sqrt x)
       f))
** warning: (expr-2:1) defmacro: defining sqrt, which is also a built-in defun
sqrt
3> (sqrt 4.0)
2.0
4> (macroexpand '(sqrt 4.0))
2.0
5> (macroexpand '(sqrt x))
(sqrt x)

However, no, (set (second x) 42) is not implemented via a macro definition for second. That would not work very well. The main reason is that it would be too much of a burden. The programmer may want to have, for a given function, a macro definition which has nothing to do with implementing assignment semantics!

Moreover, if (second x) implements place semantics, what happens when it is not embedded in an assignment operation, such that the semantics is not required at all? Basically, to hit all the requirements would require concocting a scheme for writing macros whose complexity would equal or exceed that of existing logic for handling places.

TXR Lisp does, in fact, feature a special kind of macro called a "place macro". A form is only recognized as a place macro invocation when it is used as a place. However, place macros do not implement place semantics themselves; they just do a straightforward rewrite. Place macros must expand down to a form that is recognized as a place.

Example: specify that (foo x), when used as a place, behaves as (car x):

1> (define-place-macro foo (x) ^(car ,x))
foo
2> (macroexpand '(foo a)) ;; not a macro!
(foo a)
3> (macroexpand '(set (foo a) 42)) ;; just a place macro
(sys:rplaca a 42)

If foo expanded to something which is not a place, things would fail:

4> (define-place-macro foo (x) ^(bar ,x))
foo
5> (macroexpand '(foo a))
(foo a)
6> (macroexpand '(set (foo a) 42))
** (bar a) is not an assignable place

Humphries

I ended up switching for my toys to use something that is extremely simple and works reasonably well for me... when setf sees a form as place, checks if there is a function or macro named set-x where x is the first element of the form and rewrites to an invokation of that, adding the value as last parameter. So for example by defining a regular function/macro set-aref you get (setf (aref x y) z) expanding to (set-aref x y z). I found this approach simpler than CL setf machinery. – Superstratum

@Superstratum CL has that too, except the name of the function is a list rather than a calculated symbol name. If setf doesn't find an expander for the (foo ...) form (whatever it is), then it will try calling the function (setf foo) similarly to how you have it. But of course it has all the expansion machinery too. If all you have is the setter function fallback then it's a subset of what setf does. – Humphries

Yeah... I know. That's the only part I found needed (once you have function and macros in separate namespaces). Actually my current approach is even dumber... there's only one namespace, but if the first element of a form is a symbol, say in (foo x) then first is checked if there is a function named m/foo and in that case macroexpansion is performed, otherwise the compiler considers that to be (funcall f/foo x). – Superstratum