Haskell - Parsec testing with the help of QuickCheck

Question

I'd like to write a tests for a Parsec parser. Here is the example of parser and data structure:

data Event = Event { keyEvent :: String }
    deriving Show

parseKey :: Parser Event
parseKey = do
             char '<'
             k <- many1 (letter <|> digit <|> oneOf "_")
             char '>'
             return $ Event k

I know that I need to check the parse = parse . pretty . parse property. But how should I generate the correct and incorrect test cases? Generally, how should I generate test cases for given BNF? I know about instance Arbitrary, but not much help from this.

I would appreciate if you provide a well-commented example of generator for this simple parser.

Wicket

For correct test-cases would checking id = parse . pretty be good enough? (Also, oneOf "_" == char '_') – Ingra

I think that would be enough, yeah! (Also, the first version was far more strange). – Wicket

@dbaupp: I've think about this and no, consider simple example when there is spaces or other skippable characters in parser, then, the result produced would be different. And additional parse would help. – Wicket

Answer 1

Testing parsers is not entirely trivial. (Depending on the complexity of the thing, as with all testing.)

One thing you can do is write an Arbitrary instance which constructs all valid expressions (or whatever it is you're trying to parse), and then check that if you pretty-print the thing and then parse the resulting string, you get back exactly what you started with.

There are a couple of problems with that:

• If the answer is wrong, what's broken? The parser or the printer?

• If the thing you're parsing is complicated enough to have optional brackets and things, you need to check it works both with and without the optional brackets. Your pretty printer will only do it one way or the other, usually.

• This doesn't check that garbage input really does get rejected (and not parsed as something strange). E.g., I've written plenty of Parsec parsers which will silently ignore a syntax error if it occurs at the end of the input.

In general, the only really thorough way I know of testing a parser is to just write lots and lots of manual tests by hand. (And every time you find something that parses wrong, add another test case for it.) This is essentially what GHC does with their test suit, for example.

Of course, it depends how complex your parser is, and how much assurance you want... If you're just parsing JSON, you can probably test it fairly easily. If you're parsing something like Markdown... my God have mercy on your soul!

Crossarm

I'm writting the parser of XCompose, if interested here is the sources. So, would creating QuickCheck for this is it makes sense? Or hand-written tests would be better? And what library can be proposed for second case? – Wicket

You can of course do both of these things. Use QuickCheck to verify that the round-trip from AST to string to AST works, and write some manual tests to check for edge cases. It looks like a fairly small grammar, so it shouldn't be too hard to test... – Crossarm