Find the value that failed for quickcheck

Question

When a value fails a QuickCheck'd test, I'd like to use it for debugging. Is there any way I can do something like:

let failValue = quickCheck' myTest
in someStuff failValue

If my data was readable then I could probably hack some way to get it in from IO, but it's not.

Abdel

Answer 1

I couldn't find anything in the QuickCheck API to do this in a nice way, but here's something I hacked together using the monadic QuickCheck API. It intercepts and logs the inputs to your property in an IORef, and assumes that if it failed, the last one was the culprit and returns it in a Just. If the test passed, the result is Nothing. This can probably be refined a bit but for simple one-argument properties it should do the job.

import Control.Monad
import Data.IORef
import Test.QuickCheck
import Test.QuickCheck.Monadic

prop_failIfZero :: Int -> Bool
prop_failIfZero n = n /= 0

quickCheck' :: (Arbitrary a, Show a) => (a -> Bool) -> IO (Maybe a)
quickCheck' prop = do input <- newIORef Nothing
                      result <- quickCheckWithResult args (logInput input prop)
                      case result of
                         Failure {} -> readIORef input
                         _ -> return Nothing
  where
    logInput input prop x = monadicIO $ do run $ writeIORef input (Just x)
                                           assert (prop x)
    args = stdArgs { chatty = False }

main = do failed <- quickCheck' prop_failIfZero
          case failed of
              Just x -> putStrLn $ "The input that failed was: " ++ show x
              Nothing -> putStrLn "The test passed"

Rostock

This little trick just made my Haskell debugging experience a lot better. Thanks – Tuberculosis

I was struggling myself with this for a while and that post was quite useful to me as food for thoughts, so thanks a lot for that. I thought I'd also drop in here a pointer to whenFail (and wnehFail') (hackage.haskell.org/package/QuickCheck-2.13.2/docs/…) which take an IO () to perform on failure, and ended up being what I used. – Dissuasion

Answer 2

One way would be to use the sample' method, manually run the test and find the values in which it fails. For example, testing a faulty double function:

import Test.QuickCheck

double :: Int -> Int
double x | x < 10 = 2 * x
         | otherwise = 13

doubleTest :: Int -> Bool
doubleTest x = x + x == double x

tester :: IO ()
tester = do
  values <- sample' arbitrary
  let failedValues = filter (not . doubleTest) values
  print failedValues

The only problem is sample' only generates 11 test values, that may not be enough to trigger the bug.

Po