How to test a clojure macro that uses gensyms?

Question

I want to test a macro that uses gensyms. For example, if I want to test this:

(defmacro m1
  [x f]
  `(let [x# ~x]
    (~f x#)))

I can use macro-expansion...

(macroexpand-1 '(m1 2 inc))

...to get...

(clojure.core/let [x__3289__auto__ 2] (inc x__3289__auto__))

That's easy for a person to verify as being correct.

But how can I test this in an practical, clean automated fashion? The gensym is not stable.

(Yes, I know the particular macro example is not compelling, but the question is still fair.)

I realize Clojure expressions can be treated as data (it is a homoiconic language), so I can pull apart the result like this:

(let [result (macroexpand-1 '(m1 2 inc))]
  (nth result 0) ; clojure.core/let
  (nth result 1) ; [x__3289__auto__ 2]
  ((nth result 1) 0) ; x__3289__auto__
  ((nth result 1) 0) ; 2
  (nth result 2) ; (inc x__3289__auto__)
  (nth (nth result 2) 0) ; inc
  (nth (nth result 2) 1) ; x__3289__auto__
  )

But this is unwieldy. Are there better ways? Perhaps there are data structure 'validation' libraries that could come in handy? Maybe destructuring would make this easier? Logic programming?

UPDATE / COMMENTARY:

While I appreciate the advice of experienced people who say "don't test the macro-expansion itself", it doesn't answer my question directly.

What is so bad about "unit testing" a macro by testing the macro-expansion? Testing the expansion is reasonable -- and in fact, many people test their macros that way "by hand" in the REPL -- so why not test it automatically too? I don't see a good reason to not do it. I admit that testing the macro-expansion is more brittle than testing the result, but doing the former can still have value. You can also test the functionality as well -- you can do both! This isn't an either/or decision.

Here is my psychological explanation. One of the reasons that people don't test the macro-expansion is that it currently is a bit of a pain. In general, people often rationalize against doing something when it seems difficult, independent of its intrinsic value. Yes -- that is exactly why I asked this question! If it were easy, I think people would do it more often. If it were easy, they would be less likely to rationalize by giving answers saying that "it is not worth doing."

I also understand the argument that "you should not write a complex macro". Sure. But let's hope people do not go as far as to think "if we encourage a culture of not testing macros, then that will prevent people from writing complex ones." Such an argument would be silly. If you have a complex macro-expansion, testing that it works as you expect is a sane thing to do. I am personally not beneath testing even simple things, because I am often surprised that bugs can come from simple mistakes.

Atwekk

Answer 1

Don't test how it works (its expansion), test that it works. If you test the particular expansion, you are chained to that implementation strategy; instead, just test that (m1 2 inc) returns 3, and whatever other test cases are necessary to comfort your conscience, and then you can be happy that your macro is working.

Blackthorn

Thanks for being willing to say "try a different approach". Your advice is certainly shared by many. I agree that testing the final result is a good strategy. (But that is not to say that also testing the expansion is unwise; I think both together give a more comprehensive test.) For this question, I do want to test the macro-expansion. Whether that is wise is certainly debatable (see above). – Atwekk

Answer 2

This can be done with metadata. Your macro outputs a list, which can have metadata attached to it. Simply add the gensym->var mappings to that and then use them for testing.

So your macro would look something like this:

(defmacro m1 [x f]
  (let [xsym (gensym)]
    (with-meta 
      `(let [~xsym ~x]
         (~f ~xsym))
      {:xsym xsym})))

The output from the macro now has a map against it with the gensyms:

(meta (macroexpand-1 '(m1 a b)))
=> {:xsym G__24913}

To test the macro expansion you would do something like this:

(let [out (macroexpand-1 `(m1 a b))
      xsym (:xsym (meta out))
      target `(clojure.core/let [~xsym a] (b ~xsym))]

  (= out target))

To address the question why you would want to do this: The way I normally write a macro is to generate the target code first (i.e. what I want the macro to output), test that does the right thing, then generate the macro from that. Having known-good code up front allows me to do TDD against the macro; in particular I can tweak the macro, run the tests, and if they fail clojure.test will show me the actual vs. target and I can visually inspect.

Tedmann