Have you used Quickcheck in a real project [closed]

Question

Quickcheck and its variants (even there is one in Java), seems to be interesting. However, apart from academic interest, is it really useful in a real application testing (Eg. a GUI application or Client/Server or even take StackOverflow itself)? Any experiences you had with similar test generators is appreciated.

Ogrady

Answer 1

Yes, well. Actually no, but I've studied under the man who originally developed QuickCheck and he's a really interesting guy.

Back in 2004, we were forced to use QuickCheck to test our Haskell programs and it was combination of good and bad. Mostly bad because Haskell was a bit daunting itself but never the less wonderful when you got it working.

John has since then perfected that which he wrote years back and actually helped Ericssion test their complex telecom hardware, and he found bugs in 20 millions or so lines of code reducing that to a mere three steps through his approach. He's a great speaker so it's always a joy listening to him present what he does so well, but all in all, what he did with QuickCheck was new to me. So I asked him, what his interest was in bringing this to the market. He was open to the idea, but at the time his business (based around QuickCheck) was relatively new and so there were other areas he would focus on. This is now 2007. My point is, you could learn from QuickCheck even if you wont end up using it.

But what is QuickCheck? It's a a combinatorial testing framework and an interesting way to test programs. The people over at Microsoft Research has built Pex which is sort of similar. Pex generates tests automatically by examining your IL. However, John would write a generator for possible input and test properties of a function. A property is something which can easily be tested and it's a lot more formal. e.g. reversing a list? Well, reversing a list, is the same thing as splitting a list in two halves, reversing them each individually and then concatenating the two reversed halves in reverse order.

1,2,3,4 // original
1,2 3,4 // split into A and B
2,1 4,3 // reverse A and B
4,3,2,1 // concat B and A

This is a great property to test with QuickCheck called the specification and the result is quite astonishing.

Pex is nice, but not as cool as QuickCheck, Pex simplifies things, QuickCheck does to but it takes a lot of effort to write a good specification.

The power of QuickCheck is that when it runs into a failure it will reduce the input which caused your test to fail, to the smallest possible form. Leaving you with a detailed description of what progression of state caused your test to fail. In comparison to other testing frameworks which will just try to break your code in a brute force manner.

This is made possible due to how you write your testing specification. QuickCheck relies on pseudo-randomness to invent input and it's because of this, its capable of backtracking and find really small input which does not pass your test.

It's a lot more work to write QuickCheck properties but the end result is better testing. As John himself said, 70% of bugs are caught by unit testing, but it's that other 30% which causes your program to crash. QuickCheck is testing those last 30%.

Iorgos

Very well said. However, coming back to my question, can we use it in other cases where the function may not be pure (it may just be doing things as side effects). Let say a function to copy a file from one location to another. – Ogrady

Oh yeah sure, that your function has side effects isn't really a problem. It's problematic per se but it does not prevent you from using QuickCheck. The Ericssion test suit had side effects and there's way you can get around that. – Iorgos

It does involve being careful and thinking about state progression. But the basic principle apply. The reduction part is trickier but it's basically guess work. Take bad input and remove input as long as the crash keeps happening. That is not limited to pure functions. – Iorgos

Thanks John. Looks like I should take a deeper look at Quickcheck. Thanks a lot. – Ogrady

Answer 2

I've done a real Haskell problem which involved a discrete event simulation. So I wrote a DES library based on the continuation monad, along with the equivalents to MVars and Channels. I needed to check that this worked properly, so I wrote a bunch of QuickCheck properties to demonstrate that, for instance, two streams of concurrent data written to a Channel would be correctly merged without dropping anything.

I've also used QuickCheck to document and verify the properties in my Ranged Sets and Decimal libraries.

In my experience QuickCheck is sometimes great. If you can summarise an important property in a concise way, although the algorithm that delivers that property is hairy, then QuickCheck is a huge win. On the other hand I often find that the algorithm is equivalent to the property I want to verify. In that case I look for simpler properties. For instance, suppose function "foo" is supposed to be non-strictly monotonic. Then you can write

prop_fooMonotonic x y = (x > y) ==> (foo x >= foo y)

Disarrange

Answer 3

I use QuickCheck for a lot of personal stuff. Within the last six months:

• Ran QuickCheck to test color transformations and discrete cosine transforms in an image compressor.

• Ran QuickCheck to test a symbolic-differentiation module I whipped up for some numerical optimization.

• Ran QuickCheck to test a ternary-search tree in the style of Bentley and Sedgewick.

QuickCheck rarely meets all my unit-testing needs, but it's a great way to get started---and the QuickCheck laws make good documentation.

Herrmann

Obviously QuickCheck isn't a good fit for testing code in the IO monad, but do you find that QuickCheck is a good fit for testing all pure code? If not, where is it not a good fit? – Eldrida

Answer 4

I've used it quite a lot, mostly in straighforward manner, chiefly for testing protocol and parser implementations.

However, here is less trivial bit from my personal experience: http://www.haskell.org/haskellwiki/QuickCheck_as_a_test_set_generator

Shipowner

Answer 5

ScalaCheck (a QuickCheck for Scala) is used to test Functional Java, a library that, among other things, implements a QuickCheck for Java.

Mincemeat

Answer 6

AFAIK XMonad is tested extensively with QuickCheck

Ungrounded

Answer 7

I have only used Haskell in a production environment for the development of little helper tools. Mainly because I'm the only developer I know that reads Haskell. I used QuickCheck extensively though, and got really annoyed that something similar is not available in C#. So I decided to try and write it myself. I looked at Pex too, but found the program exploration techniques that are used to find minimal input less interesting than the way QuickCheck does that.

Stockist

Answer 8

I use QuickCheck to test the behaviour of command-line programs written in any language.

It is especially useful to find inputs on which low-level or dynamically typed programs crash.

For convenience, I wrote http://hackage.haskell.org/package/proctest, which includes some examples of QuickCheck being used together with hspec and HUnit to test command line programs this way.

Marlinemarlinespike

Answer 9

We use FsCheck to check that our OCaml to F# translations are correct, and that our optimized versions work the same as the un-optimized versions. I also plan to use it to test the lexer and parser since the NHol project uses a parser combinator.

We also have helper functions that allow us to run the test within NUnit (XUnit for .Net). See assertProp.

Shiflett

Answer 10

I have used QuickCheck for testing SMS PDU encoder and decoder in LG Linux Mobile platform. A (old) version of a piece of software I developed at that time is available for download at http://hackage.haskell.org/package/GenSmsPdu-0.1

Cuxhaven

Answer 11

It's not my project, but the containers package uses QuickCheck fairly extensively. Personally, I tried using it to compare a silly little prime sieve I wrote to one in arithmoi, which led me to discover that one in arithmoi sometimes segfaults.

Diadromous