How can I re-try a property-based-test if the randomly-generated inputs are not useful?

Question

I am a n00b to unit testing. I have installed FsCheck.Nunit and NUnitTestAdapter from Nuget, and I'm trying to do property-based-testing, largely inspired by the inestimable Scott Wlaschin.

I am using the [<Property>] attribute, and I would like the ability to "skip" inputs that don't meet the test's requirements:

[<Property(MaxTest=10)>]
let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest =
    if Directory.Exists(badDest)
    then // somehow skip to the next randomized input
    else // do the actual test

What's the simplest way to do this?

I would prefer an answer for FsCheck/NUnit if it exists, but I would also consider any other framework whose tests can be run in Visual Studio. (I thought I saw some framework where there was a simple function to do exactly this, but I can't figure out what it was.)

I have preferred FsCheck.NUnit so far because it can generate random inputs for F# types (discriminated unions, etc) without additional work.

Ackler

You need a custom generator. – Gillman

IIRC for the then statement you only have to return the result that states the test succeeded without running the test. Something as simple as true might work. I ran into this years back and the problem was overthinking the problem. Remember the test is just a function and the test driver just wants a result indicating success or failure so give it a success. – Spent

@Guy Coder The rub is that I want the test to be re-run with different random inputs if the generated ones are not useful -- I don't want the function to run many times with unsuitable inputs and declare itself a success if it never actually performed its test. – Ackler

Answer 1

You should be able to do something like this:

open FsCheck
open FsCheck.Xunit

[<Property(MaxTest=10)>]
let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest =
    (not Directory.Exists(badDest)) ==> lazy
    // do the actual test

The first line is a boolean condition, and ==> is a custom operator defined by the FsCheck module. It'll only force evaluation of the lazy expression if the condition on the right-hand side evaluates to true.

Do consider, however, refactoring this test so that it doesn't depend on the file system. The file system is persistent, so that automatically creates a Persistent Fixture, which is a lot of trouble to manage; not impossible to deal with, but better avoided.

This example uses FsCheck.Xunit, but IIRC FsCheck.Nunit works the same way. You should seriously consider using FsCheck.Xunit instead of FsCheck.Nunit, though. The extensibility model of NUnit 2 is extraordinarily poor, which means that most Glue Libraries that attempt to extend NUnit come with lots of problems. This isn't a problem with FsCheck.Nunit, but with NUnit itself, but it'll manifest itself in lots of trouble for you.

Cake

Awesome, that's exactly what I was looking for. I probably saw it on your blog in the first place. Thanks for the tip on NUnit vs Xunit as well. – Ackler

Normally I try to keep tests independent of the file system, DB state, etc, but for the functions I'm developing here, manipulating the file system is their primary purpose. How else can I test them at all? – Ackler

@OverlordZurg Perhaps this answer answers that question: codereview.stackexchange.com/a/99290/3878 ... or perhaps it doesn't :) – Cake

Answer 2

FsCheck.Prop.discard() seems to do what I want -- when I run the test with logging, I can see that some attempts were discarded, but 10 runs were completed without being discarded.

The ==> operator works for running the tests with FsCheck.Quick or similar. However, that requires the lazy part to be in the format of 'Testable, where the tests I'm currently writing are just <inputs>->unit.

Ackler

"that requires the lazy part to be in the format of 'Testable" That ought not to be a problem. The compiler will infer the appropriate type for you. See here for examples: blog.ploeh.dk/2015/09/08/ad-hoc-arbitraries-with-fscheckxunit – Cake

Calling discard in your then branch, or using the approach with ==> should be completely equivalent. Though ==> is the preferred approach - it reads better imo anddiscard works by throwing and catching an exception under the hood, so may be slow and disruptive for debugging. – Amphiprostyle

Found the problem - I was trying to return unit, but I needed to return bool. Unfortunately, returning bool from the lazy part (rather than Assert.AreEqual()) loses me the nice formatting that explicitly states expected vs. actual value. I am continuing to explore options, but at least I understand now :) – Ackler

w.r.t my previous comment, the thought (eventually) occurs that I can call Assert.Whatever(), which throws an exception if not satisfied, and then just return "true" at the end to achieve the same result (helpful formatting on error). – Ackler

Answer 3

I'm not terribly familiar with F# or fscheck, but NUnit provides the Assert.Ignore() function, which will immediately stop the test and marked it as "ignored." You could also use Assert.Inconclusive() or Assert.Pass() if you felt those were more appropriate statuses.

Fireresistant

I have tried "Assert.Ignore()", but that causes tests marked with the [<FsCheck.NUnit.Property>] attribute to fail, which is not the behaviour I want. – Ackler