How to set constant seeds for Haskell's quickCheck function

Question

Every time I run "quickCheck prop_xyz", a new random seed is used. How do I enforce QuickCheck to always use the same random seed?

Thanks!

Yellowhammer

May I ask why you want to? If you want repeatable tests, a good shrink implementation is far more useful. The whole point here is to be random so a constant seeds defeats that – Enswathe

@jozefg: I know we've found it useful in the past when you want to evaluate how your generators, etc., are performing. – Muniments

@jozefg: We are running some experiments using QuickCheck and want to reproduce our results. Thanks for asking. – Yellowhammer

Answer 1

The functionality you need is in Test.QuickCheck; use quickCheckWith to specify custom Args. In particular, there's the replay :: Maybe (StdGen, Int) field, which allows you to replay tests. So you can use the stdArgs defaults and tweak them; for instance,

ghci> :load Main.hs
ghci> import Test.QuickCheck
ghci> import System.Random -- for mkStdGen
ghci> quickCheckWith stdArgs{replay = Just (mkStdGen 42, 0)} prop_xyz

The second component of the tuple has to do with the size of the test cases, but I forget exactly what.

Muniments

It's Test.QuickCheck.Random.mkQCGen in recent version(s?). – Carsick

As @SebastianGraf alludes to, you need the a Test.QuickCheck.Random.QCGen, not a System.Random.StdGen. It isn't listed on hackage, but Test.QuickCheck.Random exposes mkQCGen which serves the same purpose as mkStdGen. – Herwig

Answer 2

Use the quickCheckWith function

quickCheckWith (stdArgs{replay = Just (myNewGen, testSize)}) property

If you have a test that's failing and you want to reuse it,

result <- quickCheckResult failing_prop
let gen = usedSeed result
let size = usedSize result

to get the size and seed used in a failing test.

Since you also want reproducible errors, a good shrinking algorithm may help. By default it returns [] but provide a nice enough one, then you can end up with the same (minimal) failures even on different random runs.

Enswathe

Answer 3

Example with QuickCheck >= 2.7 and tf-random.

If the output of your failing test run looks something like this:

Failure {
    usedSeed = TFGenR 1CE4E8B15F9197B60EE70803C62388671B62D6F88720288F5339F7EC521FEBC4 0 70368744177663 46 0,
    USEDSIZE = 75,
    ...        
}

Then you can reproduce the failure as follows:

import Test.QuickCheck.Random (QCGen(..))
import System.Random.TF (TFGen)

qcheck :: Testable a => a -> IO ()
qcheck prop = quickCheckWith args prop
    where
        usedSeed = QCGen (read "TFGenR 1CE4E8B15F9197B60EE70803C62388671B62D6F88720288F5339F7EC521FEBC4 0 70368744177663 46 0"::TFGen)
        usedSize = 75
        args = stdArgs{
            replay=Just(usedSeed, usedSize)
        }

Hutcheson

Answer 4

If you want to determine the Generator that caused the failure you could do the following:

import Test.QuickCheck
import System.Random


reproducableTest :: Testable a => a -> Maybe (StdGen, Int) -> IO ()
reproducableTest prop repl = do result <- quickCheckWithResult stdArgs{replay = repl} prop
                                case result of
                                     Failure{interrupted = False} -> do putStrLn $ "Use " ++ show (usedSeed result) ++ " as the initial seed"
                                                                        putStrLn $ "Use " ++ show (usedSize result) ++ " as the initial size"
                                     _       -> return ()

Only the first of the two seed numbers should be used as an argument for the function. So if you have your property prop, your initial call would be reproducableTest prop Nothing. You will get something like

  Use x y as the initial seed
  Use z as the initial size

Then you would call again with reproducableTest prop $ Just (mkStdGen x , z)

Ashely

It would be better to use read "x y" :: StdGen, since show (mkStdGen 42) == "43 1". (read and show are guaranteed to behave properly here.) – Muniments