How to use FsCheck to generate random numbers as input for property-based testing

Question

I thought it's time to try out FsCheck but it proves tougher than I thought. There's a lot of documentation on Arb, generators and so on, but there doesn't seem to be any guidance in how to apply that knowledge. Or I'm just not getting it.

What may make it harder to grasp is that the relation between tests, properties, generators, arbitraries, shrinking and, in my case, randomness (some tests automatically generate random data, others don't) is not clear to me. I don't have a Haskell background so that doesn't help much either.

Now for the question: how do I generate random integers?

My test scenario can be explained on the properties of multiplication, let's say distributivity:

static member  ``Multiplication is distributive`` (x: int64) y z =
    let res1 = x * (y + z)
    let res2 = x * y + x * z

    res1 = res2

// run it:
[<Test>]
static member FsCheckAsUnitTest() =
    Check.One({ Config.VerboseThrowOnFailure with MaxTest = 1000 }, ``Multiplication is distributive``)

When I run this with Check.Verbose or the NUnit integration, I get test sequences like:

0:
(-1L, -1L, -1L)
1:
(-1L, -1L, 0L)
2:
(-1L, -1L, -1L)
3:
(-1L, -1L, -1L)
4:
(-1L, 0L, -1L)
5:
(1L, 0L, 2L)
6:
(-2L, 0L, -1L)
7:
(-2L, -1L, -1L)
8:
(1L, 1L, -2L)
9:
(-2L, 2L, -2L)

After 1000 tests it hasn't gotten over 100L. Somehow I imagined this would "automatically" choose random numbers evenly distributed over the whole range of int64, at least that's how I interpreted the documentation.

Since it doesn't, I started experimenting and came up with silly solutions like the following to get higher numbers:

type Generators = 
    static member arbMyRecord =
        Arb.generate<int64>
        |> Gen.where ((<) 1000L)
        |> Gen.three
        |> Arb.fromGen

But this becomes incredibly slow and is clearly not the right approach. I'm sure there must be a simple solution that I'm missing. I tried with Gen.choose(Int64.MinValue, Int64.MaxValue), but this only supports ints, not longs (but even with just ints I couldn't get it working).

In the end I need a solution that works for all the primitive numeric data types, that includes their maxes and mins, their zeroes and ones, and some random selection from whatever is within.

Llama

I think it initally limits the max. to 100, see: fscheck Q – Marxmarxian

@s952163, yes, that's why I tried with MaxTest = 1000, see code above. But that doesn't help. Perhaps you mean the StartTest and EndTest values, but setting these to Int32.MinValue/MaxValue has the effect that all permutations use Int32.MinValue as a constant value. – Llama

Answer 1

As explained in this other FsCheck question, the default configurations for most of the Check functions has EndSize = 100. You can increase that number, but you can also, as you suggest, use Gen.choose.

Even so, though, the int generator is intentionally well-behaved. It doesn't, for example, include Int32.MinValue and Int32.MaxValue, since this could lead to overflows.

FsCheck does, however, also come with generators that give you uniform distributions over their entire range: Arb.Default.DoNotSizeInt16, Arb.Default.DoNotSizeUInt64, and so on.

For floating point values, there's Arb.Default.Float32, which , according to its documentation, generates "arbitrary floats, NaN, NegativeInfinity, PositiveInfinity, Maxvalue, MinValue, Epsilon included fairly frequently".

There's no uniform API for 'just' any number, since F# doesn't have typeclasses (this is something you'd be able to express in Haskell).

Also, I'm not sure your typical unit testing framework will be able run generic tests, but at least with xUnit.net, you can use this trick to run generically typed tests.

---

Specifically, though, you can write the above test like this, using FsCheck.Xunit:

open FsCheck
open FsCheck.Xunit

[<Property>]
let ``Multiplication is distributive`` () =
    Arb.generate<DoNotSize<int64>>
    |> Gen.map (fun (DoNotSize x) -> x)
    |> Gen.three
    |> Arb.fromGen
    |> Prop.forAll <| fun (x, y, z) ->

        let res1 = x * (y + z)
        let res2 = x * y + x * z

        res1 = res2

This could hypothetically fail from overflowing, but after having run some 1,000,000 cases, I haven't seen it fail yet.

The generator, however, does indeed look like it's picking values from the full range of 64-bit integers:

> Arb.generate<DoNotSize<int64>> |> Gen.sample 1 10;;
val it : DoNotSize<int64> list =
  [DoNotSize -28197L; DoNotSize -123346460471168L; DoNotSize -28719L;
   DoNotSize -125588489564554L; DoNotSize -29241L;
   DoNotSize 7736726437182770284L; DoNotSize -2382327248148602956L;
   DoNotSize -554678787L; DoNotSize -1317194353L; DoNotSize -29668L]

Notice that even though I bind the size argument of Gen.sample to 1, it picks 'arbitrarily' large positive and negative values.

Nuptials

Thanks for the pointers and explanation. Since I'm testing behavior of algorithms that can take the full range of certain numeric types, I am interested in using that as input and any overflow or underflow would be a bug. I'm still unsure how to get three-way permutations that include all combinations of [0;1;-1;MinValue;MaxValue] and if there's a leftover from the test-count, arbitrary values in-between (just like the style of Arb.Default.Float32). Or is that exactly what Arb.Default.DoNoSitzeXXX do? – Llama

I've tried running the test as follows: Check.One({Config.VerboseThrowOnFailure with Arbitrary = [typeof<DoNotSize<int64>>] }, MyDistribTestThreeArgs) (and a variant with [t;t;t] if that's needed for the three arguments). However, either method throws "No instances found on type FsCheck.DoNotSize'1[[System.Int64....". I can't use Arb.Default... directly, the config needs a Type. I'm sure I'm missing the obvious, but what? – Llama

Oh, and Arbitrary = Arb.Default.DoNotSizeInt64().GetType() throws a similar error. Which is odd, you give it an exact type that also has a default ctor, but it still says it cannot find that very type. – Llama

@Llama Have you tried using Arb.Default.DoNotSizeInt64() |> Prop.forAll //...? I never use the Check module... – Nuptials

Could you perhaps update with an example with syntax you believe is correct? I can always take it from there, but right now I'm still stuck. – Llama

@Llama I have lots of examples on my blog; here's one article that showcases how to use various Gen combinators. Another place to start is the whole Types + Properties = Software article series. – Nuptials

Thanks, I'll try that out and post back here, add an answer, or maybe I can expand your answer with an example, so that my original question (how to get random ints?) gets an answer once I figured this out. I work best from example (and then back to theory). – Llama

(sorry for the rant, I removed that comment). I'm basically still stuck and wondering whether FsCheck is the right solution here? I currently have Arb.Default.DoNotSizeInt64().Generator |> Gen.map DoNotSize.Unwrap |> Gen.three |> Arb.fromGen, but how do I get from there to including the min/max, zeroes and ones? Still hoping for an answer, but perhaps it is simply too complex to fit in a Q&A like SO? Just say so, I can try to rephrase or split in multiple questions. – Llama

Thanks, looks like I was pretty close. It helps to see it in action and the FSI with Gen.sample... I'm surprised I didn't encounter that myself. I dare not ask for the how do I get from there to including the min/max, zeroes and ones, though I think now perhaps it is better to not try to mix in one generator both range values with fixed/constant values (having [min, max, 0, 1, -1] next to the randoms was part of what I'm after). Anyway, you have helped me a lot, I'll raise a separate question on that. – Llama

@Llama You can create a generator from specific values with Gen.elements, and you can combine two or more generators with Gen.oneof. There are example of generators and combinators here: fscheck.github.io/FsCheck/TestData.html If that's not enough, don't hesitate to ask another question. That's what Stack Overflow is for. – Nuptials

Gen.elements and Gen.oneof is what I needed. Thanks again! – Llama