What can you do with Lisp macros that you can't do with first-class functions?

Question

I think I understand Lisp macros and their role in the compilation phase.

But in Python, you can pass a function into another function

def f(filename, g):
  try:                                
     fh = open(filename, "rb") 
     g(fh)
  finally:
     close(fh) 

So, we get lazy evaluation here. What can I do with macros and not with functions as first class objects?

Gainful

Preprocessor? That's sooo sevent... er C-ish. – Alfaro

@delnan OK I have modified "pre-processor" to "compilation phase." – Gainful

What if you wanted to check every time you called your your f function if the filename you gave exists at "compile"(macro-expanding) time. You could change your f to a macro and back. If you did it without a macro it wouldn't check it until it got to f in the code. (You could check if the macro argument is a symbol or string and if its a string you can check the filename at compile time). Also you could check for all the possible values / symbols of g's and filename's at "compile" time and log them by changing f to a macro as well then changing it back when your done. – Provost

If by "compile-time" you mean "in-development", then such functionality belongs in build-tools, not in the main code. If by "compile (macro-expanding) time" you mean at startup/code-loading time, then OK, but any language supports init code for a given module/class. – Gainful

Yeah but Lisp lets you make a quick change without having to plan for it because macros and functions look like the same thing to Lisp (although Emacs can highlight macros and functions differently). What if your file loading functions are scattered around your module? Then you need to plan for it and change your application. Also Lisp allows you to build easily build a code walker that could take your scattered functions and join them up for init code as well... – Provost

A lot of these answers mention "new syntax", which is not very helpful considering Lisp's symbols can be used anywhere even if they are already keywords, at least to my primitive understanding. That's a Lisp feature, not a Lisp macro feature, and can't be compared to anything in Python. It's not "new syntax" so to speak, it's an inherent Lisp feature. – Vibratile

Answer 1

First of all Lisp has first-class functions too, so you could as well ask: "Why do I need macros in Lisp if I already have first-class functions". The answer to that is that first-class functions don't allow you to play with syntax.

On a cosmetic level, first-class functions allow you to write f(filename, some_function) or f(filename, lambda fh: fh.whatever(x)), but not f(filename, fh, fh.whatever(x)). Though arguably that's a good thing because in that last case it is a lot less clear where fh suddenly comes from.

More importantly functions can only contain code that is valid. So you can't write a higher-order function reverse_function that takes a function as an argument and executes it "in reverse", so that reverse_function(lambda: "hello world" print) would execute print "hello world". With a macro you can do this. Of course this particular example is quite silly, but this ability is enormously useful when embedding domain specific languages.

For example you couldn't implement common lisp's loop construct in python. Hell, you couldn't even implement python's for ... in construct in python if it wasn't really built-in - at least not with that syntax. Sure you could implement something like for(collection, function), but that's a lot less pretty.

Tracee

Thanks for that answer. As to the first two paragraphs: Can you think of give some non-silly, though still understandable, examples? – Gainful

@Joshua: DSLs, e.g. for embedding SQL (not in strings, in the code), are a common use case. – Alfaro

How about CASE? Common Lisp has a CASE macro (actually a few of them) that I find pretty useful. – Dansby

To abstract and hopefully clarify a bit, Lisp macros allow you to control whether (and when) arguments are evaluated; functions do not. With a function, the arguments are always evaluated when the function is invoked. With a macro, the arguments are passed to the macro as ordinary data. The macro code then decides itself if and when they should be evaluated. This is why you can't write reverse_function without macros: as a function, the argument would be evaluated (and cause an error) before the function body gets a chance to rewrite it. – Strata

Introducing binding forms can be confusing, but are needed if you want logic/constraint/symbolic equational solving in your code. More useful is using your macros to do code transformations to make the code faster via walking through your code and looking for patterns, caching, inlining, changing structure. adding compile time error checking, finding patterns to help refactor your code, and adding custom tracing/debugging info at compile time. Its easy in Lisp to write your own custom eval to do error checking / debugging / optimization and have your macro run your eval by quoting the args. – Provost

@PaulLegato Your comment should be an answer, it's what I was looking for when I arrived here. – Cohl

Answer 2

Here's Matthias Felleisen's answer from 2002 (via http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg01539.html):

I'd like to propose that there are three disciplined uses of macros:

1. data sublanguages: I can write simple looking expressions and create complex nested lists/arrays/tables with quote, unquote etc neatly dressed up with macros.

2. binding constructs: I can introduce new binding constructs with macros. That helps me get rid of lambda's and with placing things closer together that belong together. For example, one of our teachpacks contains a form
   (web-query ([last-name (string-append "Hello " first-name " what's your last name?"]) ... last-name ... first-name ...) with the obvious interaction between a program and a Web consumer implied.
   [Note: In ML you could write web-query(fn last-name => ...)string_append(...) but by golly that's a pain and an unnecessary pattern.]

3. evaluation reordering: I can introduce constructs that delay/postpone the evaluation of expressions as needed. Think of loops, new conditionals, delay/force, etc.
   [Note: In Haskell, you don't need that one.]

I understand that Lispers use macros for other reasons. In all honesty, I believe that this is partly due to compiler deficiencies, and partly due to "semantic" irregularities in the target language.

I challenge people to address all three issues when they say language X can do what macros can do.

-- Matthias

Felleisen is one of the most influential macro researchers in the field. (I don't know whether he would still agree with this message, though.)

More reading: Paul Graham's On Lisp (http://www.paulgraham.com/onlisp.html; Graham definitely doesn't agree with Felleisen that these are the only useful uses of macros), and Shriram Krishnamurthi's paper "Automata via Macros" (http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/).

Moxa

Thanks for the answer. OK, I understand your first item: Complex patterns of quote-unquote can be simplified. But can't the 2d and 3d be done using first-class functions (or lambdas)? – Gainful

@JoshuaFox You can not change the evaluation order just by the use of lambdas, because lambdas have a fixed evaluation order: arguments are evaluated first. An if clause must not evaluate all arguments first, because either the then or else part should not evaluated at all. – Cobalt

1.) Data sublanguages: use JSON i.e: [],{}, for example, (DEFUN MYADD (a b) (+ a b)) is [defun, 'myadd', 'a b', [addF, 'a', 'b']]. 2.) use strings, and maybe locals() : ( -- 3.) Use lambda x: x +1 which is a lazy add 1. Okay everything is kinda hacky, but I think the biggest real reason to use macros is for static time checks and efficiency. – Provost

3 should be unless(a == 5, lambda: print('NO!!'), lambda: print('its equal')) – Provost

binding constructs: In languages with first class dictionaries you can use strings (or symbol types) and hashmaps for binders. – Provost

Answer 3

Macros do code transformations

The macro transforms source code. A lazy evaluation does not. Imagine that you can now write functions which transform arbitrary code to arbitrary different code.

Very simple code transformations

The creation of simple language constructs is also only a very simple example. Consider your example of opening a file:

(with-open-file (stream file :direction :input)
  (do-something stream))

vs.

(call-with-stream (function do-something)
                  file
                  :direction :input)

What the macro gives me is a slightly different syntax and code structure.

Embedded language: advanced iteration constructs

Next consider a slightly different example:

(loop for i from 10 below 20 collect (sqr i))

vs.

(collect-for 10 20 (function sqr))

We can define a function COLLECT-FOR which does the same for a simple loop and has variables for start, end and a step function.

But LOOP provides a new language. The LOOP macro is a compiler for this language. This compiler can do LOOP specific optimizations and can also check the syntax at compile time for this new language. An even more powerful loop macro is ITERATE. These powerful tools on the language level now can be written as libraries without any special compiler support.

Walking the code tree in a macro and making changes

Next another simple example:

(with-slots (age name) some-person
  (print name)
  (princ " "
  (princ age))

vs. something similar:

(flet ((age (person) (slot-value person 'age))
       (name (person) (slot-value person 'name)))
   (print (name))
   (princ " ")
   (princ (age)))

The WITH-SLOTS macro causes the complete walk of the enclosed source tree and replaces the variable name with a call to (SLOT-VALUE SOME-PERSON 'name):

(progn
  (print (slot-value some-person 'name))
  (princ " "
  (princ (slot-value some-person 'age)))

In this case the macro can rewrite selected parts of the code. It understands the structure of the Lisp language and knows that name and age are variables. It also understands that in some situations name and age might not be variables and should not be rewritten. This is an application of a so-called Code Walker, a tool that can walk code trees and make changes to the code tree.

Macros can modify the compile-time environment

Another simple example, the contents of a small file:

(defmacro oneplus (x)
  (print (list 'expanding 'oneplus 'with x))
  `(1+ ,x))

(defun example (a b)
   (+ (oneplus a) (oneplus (* a b))))

In this example we are not interested in the macro ONEPLUS, but in the macro DEFMACRO itself.

What is interesting about it? In Lisp you can have a file with above contents and use the file compiler to compile that file.

;;; Compiling file /private/tmp/test.lisp ...
;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; ONEPLUS

(EXPANDING ONEPLUS SOURCE A) 
(EXPANDING ONEPLUS SOURCE (* A B)) 
; EXAMPLE
;; Processing Cross Reference Information

So we see, that the file compiler expands the use of the ONEPLUS macro.

What is special about that? There is a macro definition in the file and in the next form we already use that new macro ONEPLUS. We have never loaded the macro definition into Lisp. Somehow the compiler knows and registers the defined macro ONEPLUS and is then able to use it.

So the macro DEFMACRO registers the newly defined macro ONEPLUS in the compile-time environment, so that the compiler knows about this macro - without loading the code. The macro then can be executed at compile-time during macro expansion.

With a function we can't do that. The compiler creates code for function calls, but does not run them. But a macro can be run at compile time and add 'knowledge' to the compiler. This knowledge then is valid during the run of the compiler and partially forgotten later. DEFMACRO is a macro which executes at compile time and then informs the compile-time environment of a new macro.

Note also that the macro ONEPLUS is also run twice, since it is used twice in the file. The side effect is that it prints something. But ONEPLUS could have also other arbitrary side effects. For example it could check the enclosed source against a rule base and alert you if for example the enclosed code violates some rules (think of a style checker).

That means, that a macro, here DEFMACRO, can change the language and its environment during compilation of a file. In other languages the compiler might provide special compiler directives which will be recognized during compilation. There are many examples for such defining macros influencing the compiler: DEFUN, DEFCLASS, DEFMETHOD, ...

Macros can make the user code shorter

A typical example is the DEFSTRUCT macro for defining record-like data structures.

(defstruct person name age salary)

Above defstruct macro creates code for

• a new structure type person with three slots
• slot accessors for reading and writing the values
• a predicate to check if some object is of class person
• a make-person function to create structure objects
• a printed representation

Additionally it may:

• record the source code
• record the origin of the source code (file, editor buffer, REPL, ...)
• cross-reference the source code

The original code to define the structure is a short line. The expanded code is much longer.

The DEFSTRUCT macro does not need access to a meta-level of the language to create these various things. It just transforms a compact piece of descriptive code into the, typically longer, defining code using the typical language constructs.

Sapid

Good point, the macros can allow compile-time checking for constructs – Gainful

Macros allow "pluggable-type systems", for example, in Haskell you use Monads to be able to treat computations as "things" and thus allow the type system to catch logic and semantic errors at compile time (technically its the type tags themselves that prevent your NullPointerErrors and other Semantic errors but Monads are what make it all work). Macros are actually more powerful than Monads as they can look inside expressions, and its equivalent of making your own embedded DSL to only allow you to do certain things just like SQL/HTML/CSS are languages above C that disallow pointer exceptions. – Provost

Answer 4

Macros are expanded in a compile-time. Closures are constructed in runtime. With macros you can implement highly efficient compilers of embedded domain specific languages, and with high order functions you can only implement inefficient interpreters. That eDSL compilers may do all kinds of static checks, do whatever expensive optimisations you fancy implementing, but when you've got only runtime you can't do anything expensive.

And needless to mention that macros allows much more flexible syntax (literally, any syntax) for your eDSLs and language extensions.

See the answers to this question for more details: Collection of Great Applications and Programs using Macros

Sturtevant

OK, I understand that macros can be more efficient. Also, they can make radically new syntax like postfix instead of prefix notation. – Gainful

Answer 5

Instead of a high-level answer, here's a concrete suggestion: read Shriram's The Swine Before Perl. It shows how to develop a macro that is is doing several different things -- a specific control flow, a binding, and a data language. (Plus, you'll see how to actually do this kind of stuff.)

Isom

I think the user is asking what the motivation is for using macros when Python, for example, could handle Shriram’s use case. – Confederation

Um ... no, python can certainly not handle that, it is much more than "just having first class functions". The least that should be obvious is that Shriram's argument is how macros are useful in a context where they're expanding to a pile of first-class functions. The important point is therefore in an implementation that allows writing code that looks like an automaton, not one that you need to squint to see one. – Isom

Answer 6

Macros are most useful in languages which use the same form for data and code, because a macro treats the code as data and produces new code. A macro expansion is a just-in-time code generation, which is performed during a compilation phase before the evaluation starts. This makes it very easy to design DSLs.

In Python you can not take some code, pass it to a function, which generates new code, in order to execute the new code. In order to achieve something like macros in Python you have to generate an AST from your Python code, modify the AST and evaluate the modified AST.

This means that you can not write an if statement in Python. You can use only the existing one, but you can not modify it or write you own statements. But Lisp macros allow you to write your own statements. For example you can write a fi statement which behaves like if but takes the else part as its first argument and the then part as the second.

The following article describes the difference between macros and procedures more detailed: ftp://ftp.cs.utexas.edu/pub/garbage/cs345/schintro-v13/schintro_130.html

Cobalt

Answer 7

Firstly, what macros can do that functions cannot is all those actions that special operators can do. A macro call can expand into special operators, whereas a function call cannot. For instance, if we have a lexical variable x, then (mac x) can plausibly update its value. But (fun x) cannot.

Macros are expanded and thereby disappear, after which there are only function calls and special operators. Therefore, anything that macros can do not only can be done without macros but is ultimately done without them, every time. But that's exactly like saying that everything you can do in a higher level language can be coded in assembler. In pure theory yes (I mean , look, the compiler is in fact coding it all assembler, supplying an instant existence proof), but it's not practical.

Chelsae

Answer 8

In an example other than lisp, for example elixir, the if control flow statement is in fact a macro. The if is implemented as a function. But in order to have a cleaner more memorable syntax, it was also implemented as a macro.

if true do 1+2 end
if true, do: ( 1+2 )
if(true, do: 1+2)
if(true, [do: (1+2)])
if(true, [{:do, 1+2}])

All of the above are equivalent. But the first line is the macro implementation of if, which presumably gets expanded into the if function below.

By making if a function and accessible as a macro gives you this cool ability to put if control flows inside a parameter of another function, while preserving familiarity with other languages.

is_number(if true do 1+2 end)
is_number(if true, do: (1+2))

So I think Macros allow you to control syntax better, thus allowing you to create DSLs that standard functions cannot.

Litchfield