What is generative testing in Clojure?

Question

I came across Generative Testing in Clojure with spec notion and would like to learn about it.

Also providing some examples would be very useful.

Osman

Also found this: dev.solita.fi/2017/04/10/… – Trull

Answer 1

As introductory reading we've got the Rationale and Overview along with the Guide which should provide you with information both about the why and the how.

If you'd like a somewhat complex example, we can take the string->semantic-version function of leiningen.release:

(defn string->semantic-version [version-string]
  "Create map representing the given version string. Returns nil if the
  string does not follow guidelines setforth by Semantic Versioning 2.0.0,
  http://semver.org/"
  ;; <MajorVersion>.<MinorVersion>.<PatchVersion>[-<Qualifier>][-SNAPSHOT]
  (if-let [[_ major minor patch qualifier snapshot]
           (re-matches
            #"(\d+)\.(\d+)\.(\d+)(?:-(?!SNAPSHOT)([^\-]+))?(?:-(SNAPSHOT))?"
            version-string)]
    (->> [major minor patch]
         (map #(Integer/parseInt %))
         (zipmap [:major :minor :patch])
         (merge {:qualifier qualifier
                 :snapshot snapshot}))))

It takes a string and tries to parse it into a program-readable map representing the version number of some artifact. A spec for it could look like:

First some dependencies

(ns leiningen.core.spec.util
  (:require
   [clojure.spec           :as spec]
   [clojure.spec.gen       :as gen]
   [miner.strgen           :as strgen]
   [clojure.spec.test      :as test]
   [leiningen.release      :as release]))

then a helper macro

(defmacro stregex
  "Defines a spec which matches a string based on a given string
  regular expression. This the classical type of regex as in the
  clojure regex literal #\"\""
  [string-regex]
  `(spec/with-gen
     (spec/and string? #(re-matches ~string-regex %))
     #(strgen/string-generator ~string-regex)))

followed by a definition of a semantic version

(spec/def ::semantic-version-string
  (stregex #"(\d+)\.(\d+)\.(\d+)(-\w+)?(-SNAPSHOT)?"))

and some helper-specs

(spec/def ::non-blank-string
  (spec/and string? #(not (str/blank? %))))
(spec/def ::natural-number
  (spec/int-in 0 Integer/MAX_VALUE))

for the definition of the keys in the resulting map

(spec/def ::release/major     ::natural-number)
(spec/def ::release/minor     ::natural-number)
(spec/def ::release/patch     ::natural-number)
(spec/def ::release/qualifier ::non-blank-string)
(spec/def ::release/snapshot  #{"SNAPSHOT"})

and the map itself

(spec/def ::release/semantic-version-map
  (spec/keys :req-un [::release/major ::release/minor ::release/patch
                      ::release/qualifier ::release/snapshot]))

followed by the function spec:

(spec/fdef release/string->semantic-version
           :args (spec/cat :version-str ::release/semantic-version-string)
           :ret  ::release/semantic-version-map)

By now we can let Clojure Spec generate test data and feed it into the function itself in order to test whether it meets the constraints we've put up for it:

(test/check `release/version-map->string)
=> ({:spec #object[clojure.spec$fspec_impl$reify__14248 0x16c2555 "clojure.spec$fspec_impl$reify__14248@16c2555"],
     :clojure.spec.test.check/ret {:result true,
                                   :num-tests 1000,
                                   :seed 1491922864713},
     :sym leiningen.release/version-map->string})

This tells us that out of the 1000 test cases spec generated for us the function passed every single one.

Jetsam

Great answer. Would you be able to talk more about the need for the helper macro compared to achieving the same thing just using a function? – Pacha

The helper macro only exists because I repeat that code multiple times throughout my real world code. There's actually no point to it in the above example except perhaps for readability. – Jetsam

Answer 2

You may find it easiest to get started looking at clojure/test.check before you dive into Clojure Spec. From the project page:

(require '[clojure.test.check :as tc])
(require '[clojure.test.check.generators :as gen])
(require '[clojure.test.check.properties :as prop])

(def sort-idempotent-prop
  (prop/for-all [v (gen/vector gen/int)]
    (= (sort v) (sort (sort v)))))

(tc/quick-check 100 sort-idempotent-prop)
;; => {:result true, :num-tests 100, :seed 1382488326530}

In prose, this test reads: for all vectors of integers, v, sorting v is equal to sorting v twice.

What happens if our test fails? test.check will try and find 'smaller' inputs that still fail. This process is called shrinking. Let's see it in action:

(def prop-sorted-first-less-than-last
  (prop/for-all [v (gen/not-empty (gen/vector gen/int))]
    (let [s (sort v)]
      (< (first s) (last s)))))

(tc/quick-check 100 prop-sorted-first-less-than-last)
;; => {:result false, :failing-size 0, :num-tests 1, :fail [[3]],
       :shrunk {:total-nodes-visited 5, :depth 2, :result false,
                :smallest [[0]]}}

Vibrato

Nice Clojure/West 2017 presentation by Eric Normand on test.check: youtube.com/watch?v=r5i_OiZw6Sw – Palacios