When implementing property-based testing, when should I use an input generator over a precondition expression?

Question

When implementing property-based testing, when should I use an input generator over a precondition expression?

Are there performance considerations when selecting a particular option?

Internally, does one method inevitably use the other?

I would think that a precondition expression would take longer to execute in comparison to an input generator. Has anyone tested this?

Why would we need both?

Holliman

You should at least express what you think is correct and explain. Then others will know if they are thinking the same and can see or give a different answer. – Typescript

@Guy Coder. I elaborated some. Thx for the feedback. – Holliman

When running test my first concern is coverage of all possible conditions not how long they take. While I have not used common of the shelf test generators often, when I do I tend to find that they gave give you a false sense that the code is correct when in fact there is a condition they missed. Personally I prefer to use both when possible as sometimes one catches something the other missed. In other words I sometimes missing a case the generator will find. – Typescript

What's wrong with the default tools? Or is this a learning exercise? – Holliman

With parsers in particular the combinatorial explosion is large. I once did a calculation of test I would like to create for C++ and the number of test came out to 10**27, that is 10 followed by 27 zeros. I also try to avoid generating test based on looking at the code, because it becomes tailored to the code it is testing, but when you write the code and the test both is hard not too. Don't get me wrong, I would rather do testing with generators than not, but sometimes the rules for them are worse than the code they are testing/ – Typescript

I slightly get it. Good luck on that. =) – Holliman

I just bought the new F# 4.0 book an hour ago. I've progressed. But I'm still not hardwired yet. – Holliman

Answer 1

When you use a precondition expression (such as FsCheck's ==> operator), you're essentially throwing away data. Even if this only happens in one out of a hundred cases, you'd still be throwing away 1 input set for a normal property (because the default number of executions is 100, in FsCheck).

Throwing away one out of 100 is probably not a big deal.

Sometimes, however, you'd be throwing away a lot more data. If, for example, you want only positive numbers, you could write a precondition like x > 0, but since FsCheck generates negative numbers as well, you'd be throwing away 50 % of all values, after they have been generated. That's likely to make your tests run slower (but as always, when it comes to performance considerations: measure).

FsCheck comes with built-in generators for positive numbers for that very reason, but sometimes, you need more fine-grained control of the range of possible input values, as in this example.

If doing the FizzBuzz kata, for example, you may write your test for the FizzBuzz case like this:

[<Property(MaxFail = 2000)>]
let ``FizzBuzz.transform returns FizzBuzz`` (number : int) =
    number % 15 = 0 ==> lazy
    let actual = FizzBuzz.transform number
    let expected = "FizzBuzz"
    expected = actual

Notice the use of the MaxFail property. The reason you need it is because that precondition throws away 14 out of 15 generated candidates. By default, FsCheck will attempt 1000 candidates before it gives up, but if you throw away 14 out 15 candidates, on average you'll have only 67 values that match the precondition. Since FsCheck's default goal is to execute a property 100 times, it gives up.

As the MaxFail property implies, you can tweak the defaults. With 2000 candidates, you should expect 133 precondition matches on average.

It doesn't feel particularly efficient, though, so you can, alternatively use a custom generator:

[<Property(QuietOnSuccess = true)>]
let ``FizzBuzz.transform returns FizzBuzz`` () =
    let fiveAndThrees =
        Arb.generate<int> |> Gen.map ((*) (3 * 5)) |> Arb.fromGen
    Prop.forAll fiveAndThrees <| fun number ->

        let actual = FizzBuzz.transform number

        let expected = "FizzBuzz"
        expected = actual

This uses an ad-hoc in-line Arbitrary. This is more efficient because no data is thrown away.

My inclination is to use preconditions if it'd only throw away the occasional unmatching input. In most cases, I prefer custom generators.

Tabescent

Unit testing Index - http://blog.ploeh.dk/tags/#Unit Testing-ref for Mark's blogs. For some reason it will not let me make this a proper link in this comment. – Typescript