Custom FsCheck Arbitrary type broken in Xunit but working in LINQPad and regular F# program

Question

I'm trying to implement a custom Arbitrary that generates glob syntax patterns like a*c?. I think my implementation is correct, it's just that, when running the test with Xunit, FsCheck doesn't seem to be using the custom arbitrary Pattern to generate the test data. When I use LINQPad however everything works as expected. Here's the code:

open Xunit
open FsCheck

type Pattern = Pattern of string with
    static member op_Explicit(Pattern s) = s

type MyArbitraries =
    static member Pattern() = 
        (['a'..'c']@['?'; '*'])
        |> Gen.elements
        |> Gen.nonEmptyListOf 
        |> Gen.map (List.map string >> List.fold (+) "")
        |> Arb.fromGen
        |> Arb.convert Pattern string

Arb.register<MyArbitraries>() |> ignore

[<Fact>]
let test () = 
    let prop (Pattern p) = p.Length = 0
    Check.QuickThrowOnFailure prop

This is the output:

Falsifiable, after 2 tests (0 shrinks) (StdGen (1884571966,296370531)): Original: Pattern null with exception: System.NullReferenceException ...

And here is the code I'm running in LINQPad along with the output:

open FsCheck

type Pattern = Pattern of string with
    static member op_Explicit(Pattern s) = s

type MyArbitraries =
    static member Pattern() = 
        (['a'..'c']@['?'; '*'])
        |> Gen.elements
        |> Gen.nonEmptyListOf 
        |> Gen.map (List.map string >> List.fold (+) "")
        |> Arb.fromGen
        |> Arb.convert Pattern string

Arb.register<MyArbitraries>() |> ignore

let prop (Pattern p) = p.Length = 0
Check.Quick prop

Falsifiable, after 1 test (0 shrinks) (StdGen (1148389153,296370531)): Original: Pattern "a*"

As you can see FsCheck generates a null value for the Pattern in the Xunit test although I'm using Gen.elements and Gen.nonEmptyListOf to control the test data. Also, when I run it a couple times, I'm seeing test patterns that are out of the specified character range. In LINQPad those patterns are generated correctly. I also tested the same with a regular F# console application in Visual Studio 2017 and there the custom Arbitrary works as expected as well.

What is going wrong? Is FsCheck falling back to the default string Arbitrary when running in Xunit?

You can clone this repo to see for yourself: https://github.com/bert2/GlobMatcher

(I don't want to use Prop.forAll, because each test will have multiple custom Arbitrarys and Prop.forAll doesn't go well with that. As far as I know I can only tuple them up, because the F# version of Prop.forAll only accepts a single Arbitrary.)

Karee

Answer 1

Don't use Arb.register. This method mutates global state, and due to the built-in parallelism support in xUnit.net 2, it's undetermined when it runs.

If you don't want to use the FsCheck.Xunit Glue Library, you can use Prop.forAll, which works like this:

[<Fact>]
let test () = 
    let prop (Pattern p) = p.Length = 0
    Check.QuickThrowOnFailure (Prop.forAll (MyArbitraries.Pattern()) prop)

(I'm writing this partially from memory, so I may have made some small syntax mistakes, but hopefully, this should give you an idea on how to proceed.)

---

If, on the other hand, you choose to use FsCheck.Xunit, you can register your custom Arbitraries in a Property annotation, like this:

[<Property(Arbitrary = [|typeof<MyArbitraries>|])>]
let test (Pattern p) = p.Length = 0

As you can see, this takes care of much of the boilerplate; you don't even have to call Check.QuickThrowOnFailure.

The Arbitrary property takes an array of types, so when you have more than one, this still works.

If you need to write many properties with the same array of Arbitraries, you can create your own custom attributes that derives from the [<Property>] attribute. Here's an example:

type Letters =
    static member Char() =
        Arb.Default.Char()
        |> Arb.filter (fun c -> 'A' <= c && c <= 'Z')

type DiamondPropertyAttribute() =
    inherit PropertyAttribute(
        Arbitrary = [| typeof<Letters> |],
        QuietOnSuccess = true)

[<DiamondProperty>]
let ``Diamond is non-empty`` (letter : char) =
    let actual = Diamond.make letter
    not (String.IsNullOrWhiteSpace actual)

All that said, I'm not too fond of 'registering' Arbitraries like this. I much prefer using the combinator library, because it's type-safe, which this whole type-based mechanism isn't.

Votaw

Using the Property attribute to register MyArbitraries did the trick. I wasn't using the combinators, because each test needs multiple custom arbitraries, but Prop.forAll only accepts one. At least in F#; the C# version Prop.ForAll has overloads for multiple arbitraries. Also I like the way how FsCheck.Xunit.Propertys use the test function's parameters to create the needed generators. IMO that's more readable than using Prop.forAll. – Karee

Just checked the source of Prop.forAllFunc2Bool and realized I'm stupid. Of course you can always nest calls to Prop.forAll using their property lambdas. All Arbitrarys are then available inside the deepest lambda. – Karee

Ok, I did some more digging and it turns out that the gen computation expression is indeed the most clean and extensible solution for my problem. – Karee