Using QuickCheck to test intentional error conditions

Question

I've seen how QuickCheck can be used to test monadic and non-monadic code, but how can I use it to test code that handles errors, i.e., prints some message and then calls exitWith?

Marcionism

Answer 1

A disclaimer first: I'm not an expert on QuickCheck and I had no experience with monadic checking before your question, but I see stackoverflow as an opportunity to learn new things. If there's an expert answer saying this can be done better, I'll remove mine.

Say you have a function test that can throw exceptions using exitWith. Here's how I think you can test it. The key function is protect, which catches the exception and converts it to something you can test against.

import System.Exit
import Test.QuickCheck
import Test.QuickCheck.Property
import Test.QuickCheck.Monadic

test :: Int -> IO Int
test n | n > 100   = do exitWith $ ExitFailure 1
       | otherwise = do print n
                        return n

purifyException :: (a -> IO b) -> a -> IO (Maybe b)
purifyException f x = protect (const Nothing) $ return . Just =<< f x

testProp :: Property
testProp = monadicIO $ do
  input <- pick arbitrary
  result <- run $ purifyException test $ input
  assert $ if input <= 100 then result == Just input
                           else result == Nothing

There are two disadvantages to this, as far as I can see, but I found no way over them.

1. I found no way to extract the ExitCode exception from the AnException that protect can handle. Therefore, all exit codes are treated the same here (they are mapped to Nothing). I would have liked to have:

   purifyException :: (a -> IO b) -> a -> IO (Either a ExitCode)
   

2. I found no way to test the I/O behavior of test. Suppose test was:

   test :: IO ()
   test = do
     n <- readLn
     if n > 100 then exitWith $ ExitFailure 1
                else print n
   

   Then how would you test it?

I'd appreciate more expert answers too.

Pashto

Answer 2

The QuickCheck expectFailure function can be used to handle this type of thing. Take this simple (and not-recommended) error-handling framework:

import System.Exit
import Test.QuickCheck
import Test.QuickCheck.Monadic

handle :: Either a b -> IO b
handle (Left _)  = putStrLn "exception!" >> exitWith (ExitFailure 1)
handle (Right x) = return x

and whip up a couple of dummy functions:

positive :: Int -> Either String Int
positive x | x > 0     = Right x
           | otherwise = Left "not positive"

negative :: Int -> Either String Int
negative x | x < 0     = Right x
           | otherwise = Left "not negative"

Now we can test some properties of the error handling. First, Right values should not result in exceptions:

prop_returnsHandledProperly (Positive x) = monadicIO $ do
  noErr <- run $ handle (positive x)
  assert $ noErr == x

-- Main*> quickCheck prop_returnsHandledProperly
-- +++ OK, passed 100 tests.

Lefts should result in exceptions. Notice the expectFailure tacked on to the start:

prop_handlesExitProperly (Positive x) = expectFailure . monadicIO $
  run $ handle (negative x)

-- Main*> quickCheck prop_handlesExitProperly
-- +++ OK, failed as expected. Exception: 'exitWith: invalid argument (ExitFailure 0)' (after 1 test):

Grandchild

Right, but this seems to expect all kinds of failures. Can we isolate those generated by exitWith calls? Can we assert what the failure code will be? – Pashto

@Pashto You can add some extra plumbing to the error handling itself if you want to test properties about exit codes. I.e. have a determineCode :: Either a b -> ExitCode function that exitWith calls, and make assertions about the behaviour of that. Don't know if it's possible to go further. – Grandchild

expectFailure seems to pass if any test fails, as opposed to if all tests fail, which is more what I'm looking for. – Marcionism