I'd like to write a Clojure with-test-tags macro that wraps a bunch of forms, and adds some metadata to the name of each deftest form - specifically, add some stuff to a :tags key, so that I can play with a tool to run tests with a specific tag.

One obvious implementation for with-test-tags is to walk the entire body recursively, modifying each deftest form as I find it. But I've been reading Let Over Lambda recently, and he makes a good point: instead of walking the code yourself, just wrap the code in a macrolet and let the compiler walk it for you. Something like:

(defmacro with-test-tags [tags & body]
  `(macrolet [(~'deftest [name# & more#]
                `(~'~'deftest ~(vary-meta name# update-in [:tags] (fnil into []) ~tags)
                   ~@more#))]
     (do ~@body)))

(with-test-tags [:a :b] 
  (deftest x (...do tests...)))

This has the obvious problem, though, that the deftest macro continues to expand recursively forever. I could make it expand to clojure.test/deftest instead, thus avoiding any further recursive expansions, but then I can't usefully nest instances of with-test-tags to label sub-groups of tests.

At this point, especially for something as simple as deftest, it looks like walking the code myself will be simpler. But I wonder if anyone knows a technique for writing a macro which "slightly modifies" certain subexpressions, without recursing forever.

For the curious: I considered some other approaches, such as having a compile-time binding-able var that I set as I go up and down the code, and using that var when I finally see a deftest, but since each macro only returns a single expansion its bindings won't be in place for the next call to macroexpand.

Edit

I did the postwalk implementation just now, and while it works it doesn't respect special forms such as quote - it expands inside of those as well.

(defmacro with-test-tags [tags & body]
  (cons `do
        (postwalk (fn [form]
                    (if (and (seq? form)
                             (symbol? (first form))
                             (= "deftest" (name (first form))))
                      (seq (update-in (vec form) [1]
                                      vary-meta update-in [:tags] (fnil into []) tags))
                      form))
                  body)))

(Also, sorry for possible noise on the common-lisp tag – I thought you might be able to help out with weirder macro stuff even with minimal Clojure experience.)

(This is a new approach, eval- and binding-free. As discussed in the comments on this answer, the use of eval is problematic because it prevents tests from closing over the lexical environments they seem to be defined in (so (let [x 1] (deftest easy (is (= x 1)))) no longer works). I leave the original approach in the bottom half of the answer, below the horizontal rule.)

The `macrolet` approach

Implementation

Tested with Clojure 1.3.0-beta2; it should probably work with 1.2.x as well.

(ns deftest-magic.core
  (:use [clojure.tools.macro :only [macrolet]]))

(defmacro with-test-tags [tags & body]
  (let [deftest-decl
        (list 'deftest ['name '& 'body]
              (list 'let ['n `(vary-meta ~'name update-in [:tags]
                                         (fnil into #{}) ~tags)
                          'form `(list* '~'clojure.test/deftest ~'n ~'body)]
                    'form))
        with-test-tags-decl
        (list 'with-test-tags ['tags '& 'body]
              `(list* '~'deftest-magic.core/with-test-tags
                      (into ~tags ~'tags) ~'body))]
    `(macrolet [~deftest-decl
                ~with-test-tags-decl]
       ~@body)))

Usage

...is best demonstrated with a suite of (passing) tests:

(ns deftest-magic.test.core
  (:use [deftest-magic.core :only [with-test-tags]])
  (:use clojure.test))

;; defines a test with no tags attached:
(deftest plain-deftest
  (is (= :foo :foo)))

(with-test-tags #{:foo}

  ;; this test will be tagged #{:foo}:
  (deftest foo
    (is true))

  (with-test-tags #{:bar}

    ;; this test will be tagged #{:foo :bar}:
    (deftest foo-bar
      (is true))))

;; confirming the claims made in the comments above:
(deftest test-tags
  (let [plaintest-tags (:tags (meta #'plain-deftest))]
    (is (or (nil? plaintest-tags) (empty? plaintest-tags))))
  (is (= #{:foo} (:tags (meta #'foo))))
  (is (= #{:foo :bar} (:tags (meta #'foo-bar)))))

;; tests can be closures:
(let [x 1]
  (deftest lexical-bindings-no-tags
    (is (= x 1))))

;; this works inside with-test-args as well:
(with-test-tags #{:foo}
  (let [x 1]
    (deftest easy (is true))
    (deftest lexical-bindings-with-tags
      (is (= #{:foo} (:tags (meta #'easy))))
      (is (= x 1)))))

Design notes:

We want to make the macrolet-based design described in the question text work. We care about being able to nest with-test-tags and preserving the possibility of defining tests whose bodies close over the lexical environments they are defined in.
We will be macroletting deftest to expand to a clojure.test/deftest form with appropriate metadata attached to the test's name. The important part here is that with-test-tags injects the appropriate tag set right into the definition of the custom local deftest inside the macrolet form; once the compiler gets around to expanding the deftest forms, the tag sets will have been hardwired into the code.
If we left it at that, tests defined inside a nested with-test-tags would only get tagged with the tags passed to the innermost with-test-tags form. Thus we have with-test-tags also macrolet the symbol with-test-tags itself behaving much like the local deftest: it expands to a call to the top-level with-test-tags macro with the appropriate tags injected into the tagset.
The intention is that the inner with-test-tags form in
```
(with-test-tags #{:foo}
  (with-test-tags #{:bar}
    ...))
```
expand to (deftest-magic.core/with-test-tags #{:foo :bar} ...) (if indeed deftest-magic.core is the namespace with-test-tags is defined in). This form immediately expands into the familiar macrolet form, with the deftest and with-test-tags symbols locally bound to macros with the correct tag sets hardwired inside them.

(The original answer updated with some notes on the design, some rephrasing and reformatting etc. The code is unchanged.)

The `binding` + `eval` approach.

(See also https://gist.github.com/1185513 for a version additionally using macrolet to avoid a custom top-level deftest.)

Implementation

The following is tested to work with Clojure 1.3.0-beta2; with the ^:dynamic part removed, it should work with 1.2:

(ns deftest-magic.core)

(def ^:dynamic *tags* #{})

(defmacro with-test-tags [tags & body]
  `(binding [*tags* (into *tags* ~tags)]
     ~@body))

(defmacro deftest [name & body]
  `(let [n# (vary-meta '~name update-in [:tags] (fnil into #{}) *tags*)
         form# (list* 'clojure.test/deftest n# '~body)]
     (eval form#)))

Usage

(ns example.core
  (:use [clojure.test :exclude [deftest]])
  (:use [deftest-magic.core :only [with-test-tags deftest]]))

;; defines a test with an empty set of tags:
(deftest no-tags
  (is true))

(with-test-tags #{:foo}

  ;; this test will be tagged #{:foo}:
  (deftest foo
    (is true))

  (with-test-tags #{:bar}

    ;; this test will be tagged #{:foo :bar}:
    (deftest foo-bar
      (is true))))

Design notes

I think that on this occasion a judicious use of eval leads to a useful solution. The basic design (based on the "binding-able Var" idea) has three components:

A dynamically bindable Var -- *tags* -- which is bound at compile time to a set of tags to be used by deftest forms to decorate the tests being defined. We add no tags by default, so its initial value is #{}.
A with-test-tags macro which installs an appropriate for *tags*.
A custom deftest macro which expands to a let form resembling this (the following is the expansion, slightly simplified for clarity):
```
(let [n    (vary-meta '<NAME> update-in [:tags] (fnil into #{}) *tags*)
      form (list* 'clojure.test/deftest n '<BODY>)]
  (eval form))
```
<NAME> and <BODY> are the arguments given to the custom deftest, inserted in the appropriate spots through unquoting the appropriate parts of the syntax-quoted expansion template.

Thus the expansion of the custom deftest is a let form in which, first, the name of the new test is prepared by decorating the given symbol with the :tags metadata; then a clojure.test/deftest form using this decorated name is constructed; and finally the latter form is handed to eval.

The key point here is that the (eval form) expressions here are evaluated whenever the namespace their contained in is AOT-compiled or required for the first time in the lifetime of the JVM running this code. This is exactly the same as the (println "asdf") in a top-level (def asdf (println "asdf")), which will print asdf whenever the namespace is AOT-compiled or required for the first time; in fact, a top-level (println "asdf") acts similarly.

This is explained by noting that compilation, in Clojure, is just evaluation of all top-level forms. In (binding [...] (deftest ...), binding is the top-level form, but it only returns when deftest does, and our custom deftest expands to a form which returns when eval does. (On the other hand, the way require executes top-level code in already-compiled namespaces -- so that if you have (def t (System/currentTimeMillis)) in your code, the value of t will depend on when you require your namespace rather than on when it was compiled, as can be determined by experimenting with AOT-compiled code -- is just the way Clojure works. Use read-eval if you want actual constants embedded in code.)

In effect, the custom deftest runs the compiler (through eval) at the run-time-at-compile-time of macro expansion. Fun.

Finally, when a deftest form is put inside a with-test-tags form, the form of (eval form) will have been prepared with the bindings installed by with-test-tags in place. Thus the test being defined will be decorated with the appropriate set of tags.

At the REPL

user=> (use 'deftest-magic.core '[clojure.test :exclude [deftest]])
nil
user=> (with-test-tags #{:foo}
         (deftest foo (is true))
         (with-test-tags #{:bar}
           (deftest foo-bar (is true))))
#'user/foo-bar
user=> (meta #'foo)
{:ns #<Namespace user>,
 :name foo,
 :file "NO_SOURCE_PATH",
 :line 2,
 :test #<user$fn__90 user$fn__90@50903025>,
 :tags #{:foo}}                                         ; <= note the tags
user=> (meta #'foo-bar)
{:ns #<Namespace user>,
 :name foo-bar,
 :file "NO_SOURCE_PATH",
 :line 2,
 :test #<user$fn__94 user$fn__94@368b1a4f>,
 :tags #{:foo :bar}}                                    ; <= likewise
user=> (deftest quux (is true))
#'user/quux
user=> (meta #'quux)
{:ns #<Namespace user>,
 :name quux,
 :file "NO_SOURCE_PATH",
 :line 5,
 :test #<user$fn__106 user$fn__106@b7c96a9>,
 :tags #{}}                                             ; <= no tags works too

And just to be sure working tests are being defined...

user=> (run-tests 'user)

Testing user

Ran 3 tests containing 3 assertions.
0 failures, 0 errors.
{:type :summary, :pass 3, :test 3, :error 0, :fail 0}

Edit

The `macrolet` approach

Implementation

Usage

Design notes:

The `binding` + `eval` approach.

Implementation

Usage

Design notes

At the REPL

Recommended topics

Hot tags

Edit

The macrolet approach

Implementation

Usage

Design notes:

The binding + eval approach.

Implementation

Usage

Design notes

At the REPL

Recommended topics

Hot tags

The `macrolet` approach

The `binding` + `eval` approach.