How to display a reason of a failed test property with quickcheck?

Question

What is the best practice to display reasons for a failed property test when it is tested via QuickCheck?

Consider for example:

prop a b = res /= []
   where
      (res, reason) = checkCode a b

Then the a session could look like:

> quickCheck prop
Falsifiable, after 48 tests:
42
23

But for debugging it would be really convenient to show the reason for failure as part of the quickCheck falsifable report.

I have hacked it like this:

prop a b = if res /= [] then traceShow reason False else True
   where
      (res, reason) = checkCode a b

Is there is a better/nicer or more quickcheckish way to do it?

Eufemiaeugen

Answer 1

I assume that your "reason" variable contains some kind of test-specific data on what went wrong. You could instead return a "Result", which contains both success/fail/invalid conditions and a string explaining what went wrong. Properties that return Results are handled by QuickCheck in exactly the same way as properties that return Bool.

(edit) Like this:

module QtTest where 

import Test.QuickCheck
import Test.QuickCheck.Property as P


-- Always return success
prop_one :: Integer -> P.Result
prop_one _ = MkResult (Just True) True "always succeeds" False [] []


-- Always return failure
prop_two :: Integer -> P.Result
prop_two n = MkResult (Just False) True ("always fails: n = " ++ show n) False [] []

Note that it is the "Result" type defined in Test.QuickCheck.Property you want.

There are also some combinators defined in Test.QuickCheck.Property which help you compose the Result rather than calling the constructor directly, such as

prop_three :: Integer -> Property
prop_three n = printTestCase ("always fails: n = " ++ show n) False

I guess it would be better style to use those.

Villiers

Can you give a simple example how to exactly return a Result such that the "reason" variable (assume it is some string or show-able value) is displayed in case of failure? – Eufemiaeugen

Thanks for the update. I was too fixated on cse.chalmers.se/~rjmh/QuickCheck/manual.html and did not look in the up to date and comprehensive module docs hackage.haskell.org/packages/archive/QuickCheck/2.4.0.1/doc/… - it seems that printTestCase is a recent addition - quickCheck 2.1 does not include it. – Eufemiaeugen

Answer 2

This works in the same way as Paul Johnson's answer but is more concise and robust to changes in MkResult:

import Test.QuickCheck.Property (succeeded, failed, reason)

prop a b =
  if res /= []
    then succeeded
    else failed { reason = reason }
   where
      (res, reason) = checkCode a b

Cabinetwork

Thanks, that's very simple and easy to use. Lets me see exactly what went wrong when I have a big setup before testing multiple properties – Bureaucratize

Answer 3

Because QuickCheck gives you the inputs to the function, and because the code under test is pure (it is, right?), you can just feed those inputs to the function and get the result. This is more flexible, because with those inputs you can also repeatedly test with tweaks to the original function until it's correct.

Petepetechia

Well, the point of the question is convenience. Successful testing is all about automating as much as possible. Having to open a ghci session and recomputing a potential expensive function does not make sense. I mean, QuickCheck already provides collect, classify to be able to enrich the output of non-failed property tests. Such enrichment is optional and does not decrease the flexibility at all. – Eufemiaeugen

I mean, there's also a practical reason for this: QuickCheck properties can be arbitrarily complex, so it's not necessarily obvious what the "output" of any given test is: you'll need to massage your code into a form that says "hey, here are the intermediate values I'm interested in!" Assuming you are willing to do this rewriting, though, there's no reason why you couldn't provide this capability, though IIRC QuickCheck doesn't have something baked in for this purpose. – Petepetechia

Answer 4

This is my solution (I use counterexample instead of printTestCase since the later one is deprecated now):

(<?>) :: (Testable p) => p -> String -> Property
(<?>) = flip (Test.QuickCheck.counterexample . ("Extra Info: " ++))
infixl 2 <?>

Usage:

main :: IO ()
main = hspec $ do
  describe "math" $ do
    prop "sum-of-square-le-square-of-sum" $ do
      \(x :: Int) (y :: Int) ->
        x * x + y * y <= (x + y) * (x + y) <?> show (x * x, y * y, x + y)

So when a test case fails, you can see something like:

   *** Failed! Falsifiable, Falsifiable (after 2 tests):
   1
   -1
   Extra Info: (1,1,0)

You can also use <?> together with .&&., .||., === and ==> etc.:

  describe "math" $ do
    prop "sum-of-square-le-square-of-sum" $ do
      \(x :: Int) (y :: Int) ->
        x * x + y * y <= (x + y) * (x + y) <?> show (x * x, y * y, x + y) .||. (1==0) <?> "haha"

Palish