How can I use my specs for their intended purposes if they are in a separate namespace?

Question

One of the examples in the clojure.spec Guide is a simple option-parsing spec:

(require '[clojure.spec :as s])

(s/def ::config
  (s/* (s/cat :prop string?
              :val (s/alt :s string? :b boolean?))))

(s/conform ::config ["-server" "foo" "-verbose" true "-user" "joe"])
;;=> [{:prop "-server", :val [:s "foo"]}
;;    {:prop "-verbose", :val [:b true]}
;;    {:prop "-user", :val [:s "joe"]}]

Later, in the validation section, a function is defined that internally conforms its input using this spec:

(defn- set-config [prop val]
  (println "set" prop val))

(defn configure [input]
  (let [parsed (s/conform ::config input)]
    (if (= parsed ::s/invalid)
      (throw (ex-info "Invalid input" (s/explain-data ::config input)))
      (doseq [{prop :prop [_ val] :val} parsed]
        (set-config (subs prop 1) val)))))

(configure ["-server" "foo" "-verbose" true "-user" "joe"])
;; set server foo
;; set verbose true
;; set user joe
;;=> nil

Since the guide is meant to be easy to follow from the REPL, all of this code is evaluated in the same namespace. In this answer, though, @levand recommends putting specs in separate namespaces:

I usually put specs in their own namespace, alongside the namespace that they are describing.

This would break the usage of ::config above, but that problem can be remedied:

It is preferable for spec key names to be in the namespace of the code, however, not the namespace of the spec. This is still easy to do by using a namespace alias on the keyword:

(ns my.app.foo.specs
  (:require [my.app.foo :as f]))

(s/def ::f/name string?)

He goes on to explain that specs and implementations could be put in the same namespace, but it wouldn't be ideal:

While I certainly could put them right alongside the spec'd code in the same file, that hurts readability IMO.

However, I'm having trouble seeing how this can work with destructuring. As an example, I put together a little Boot project with the above code translated into multiple namespaces.

boot.properties:

BOOT_CLOJURE_VERSION=1.9.0-alpha7

src/example/core.clj:

(ns example.core
  (:require [clojure.spec :as s]))

(defn- set-config [prop val]
  (println "set" prop val))

(defn configure [input]
  (let [parsed (s/conform ::config input)]
    (if (= parsed ::s/invalid)
      (throw (ex-info "Invalid input" (s/explain-data ::config input)))
      (doseq [{prop :prop [_ val] :val} parsed]
        (set-config (subs prop 1) val)))))

src/example/spec.clj:

(ns example.spec
  (:require [clojure.spec :as s]
            [example.core :as core]))

(s/def ::core/config
  (s/* (s/cat :prop string?
              :val (s/alt :s string? :b boolean?))))

build.boot:

(set-env! :source-paths #{"src"})

(require '[example.core :as core])

(deftask run []
  (with-pass-thru _
    (core/configure ["-server" "foo" "-verbose" true "-user" "joe"])))

But of course, when I actually run this, I get an error:

$ boot run
clojure.lang.ExceptionInfo: Unable to resolve spec: :example.core/config

I could fix this problem by adding (require 'example.spec) to build.boot, but that's ugly and error-prone, and will only become more so as my number of spec namespaces increases. I can't require the spec namespace from the implementation namespace, for several reasons. Here's an example that uses fdef.

boot.properties:

BOOT_CLOJURE_VERSION=1.9.0-alpha7

src/example/spec.clj:

(ns example.spec
  (:require [clojure.spec :as s]))

(alias 'core 'example.core)

(s/fdef core/divisible?
  :args (s/cat :x integer? :y (s/and integer? (complement zero?)))
  :ret boolean?)

(s/fdef core/prime?
  :args (s/cat :x integer?)
  :ret boolean?)

(s/fdef core/factor
  :args (s/cat :x (s/and integer? pos?))
  :ret (s/map-of (s/and integer? core/prime?) (s/and integer? pos?))
  :fn #(== (-> % :args :x) (apply * (for [[a b] (:ret %)] (Math/pow a b)))))

src/example/core.clj:

(ns example.core
  (:require [example.spec]))

(defn divisible? [x y]
  (zero? (rem x y)))

(defn prime? [x]
  (and (< 1 x)
       (not-any? (partial divisible? x)
                 (range 2 (inc (Math/floor (Math/sqrt x)))))))

(defn factor [x]
  (loop [x x y 2 factors {}]
    (let [add #(update factors % (fnil inc 0))]
      (cond
        (< x 2) factors
        (< x (* y y)) (add x)
        (divisible? x y) (recur (/ x y) y (add y))
        :else (recur x (inc y) factors)))))

build.boot:

(set-env!
 :source-paths #{"src"}
 :dependencies '[[org.clojure/test.check "0.9.0" :scope "test"]])

(require '[clojure.spec.test :as stest]
         '[example.core :as core])

(deftask run []
  (with-pass-thru _
    (prn (stest/run-all-tests))))

The first problem is the most obvious:

$ boot run
clojure.lang.ExceptionInfo: No such var: core/prime?
    data: {:file "example/spec.clj", :line 16}
java.lang.RuntimeException: No such var: core/prime?

In my spec for factor, I want to use my prime? predicate to validate the returned factors. The cool thing about this factor spec is that, assuming prime? is correct, it both completely documents the factor function and eliminates the need for me to write any other tests for that function. But if you think that's just too cool, you can replace it with pos? or something.

Unsurprisingly, though, you'll still get an error when you try boot run again, this time complaining that the :args spec for either #'example.core/divisible? or #'example.core/prime? or #'example.core/factor (whichever it happens to try first) is missing. This is because, regardless of whether you alias a namespace or not, fdef won't use that alias unless the symbol you give it names a var that already exists. If the var doesn't exist, the symbol doesn't get expanded. (For even more fun, remove the :as core from build.boot and see what happens.)

If you want to keep that alias, you need to remove the (:require [example.spec]) from example.core and add a (require 'example.spec) to build.boot. Of course, that require needs to come after the one for example.core, or it won't work. And at that point, why not just put the require directly into example.spec?

All of these problems would be solved by putting the specs in the same file as the implementations. So, should I really put specs in separate namespaces from implementations? If so, how can the problems I've detailed above be solved?

Mantling

You make a great case for why it is preferable to have the spec in the same namespace when using destructuring. It seems impossible to avoid the tradeoff of gaining a more precise interface at the cost of cluttering the code, but it would be great if there were... so I hope someone can answer this :) – Inheritance

I believe the intended practice is to require example.spec in example.core and just alias example.core in example.spec instead of requiring it... – Coir

@LeonGrapenthin That doesn't work; see my latest edit. – Mantling

@SamEstep In the case of prime? it boils down to a regular cyclic dependency problem - You can't use vars of ns A in ns B if you also use ns B in ns A irregardless of whether you are doing it to write specs. I believe that the resolve problem you have mentioned has been addressed (but not solved) per this commit: github.com/clojure/clojure/commit/… (only on master now) - Reading from the source it should throw an error in your case "Unable to resolve..." – Coir

@SamEstep What you could try is to use the fully qualified ns for your fdefs and not require or alias example.core at all - Alternatively one might argue that if your code depends on the parser of a spec, the spec becomes an artifact of the code and as such should go into the code directly. – Coir

@LeonGrapenthin From what I've read in the clojure.spec rationale and guide, it seems like my use case is perfectly valid and intended. – Mantling

@SamEstep I don't really see why you and levand wants to put the spec into a seperate ns at all. In his answer he makes no argument why one should do so and the more I look at the issue you have elaborated here I believe that without some new feature his answer won't hold. – Coir

@LeonGrapenthin That's exactly my point. As far as I can tell, it makes far more sense to put the specs and the functions into the same namespace. I posted this question because I saw that levand's answer had so many upvotes, tried actually putting into practice, saw that it didn't seem to work, and wanted to see if I was missing something. – Mantling

@SamEstep I re-read his answer and the argument he makes is readbility IHO. I have commented under his answer and illustrated another workflow that enables file separation without namespace separation. – Coir

Answer 1

This question demonstrates an important distinction between specs used within an application and specs used to test the application.

Specs used within the app to conform or validate input — like :example.core/config here — are part of the application code. They may be in the same file where they are used or in a separate file. In the latter case, the application code must :require the specs, just like any other code.

Specs used as tests are loaded after the code they specify. These are your fdefs and generators. You can put these in a separate namespace from the code — even in a separate directory, not packaged with your application — and they will :require the code.

It's possible you have some predicates or utility functions that are used by both kinds of specs. These would go in a separate namespace all of their own.

Pitchblack