What unit testing frameworks are available for F#

Question

I am looking specifically for frameworks that allow me to take advantage of unique features of the language. I am aware of FsUnit. Would you recommend something else, and why?

Safford

See also some suggestions here: #1469272 – Beastings

No time for extensive answer, but IIRC you can use Pex with F# too (I even asked a question about it once). My current favourite is Xunit. – Amply

Are you saying Pex can generate F# code? Can you provide a link? Initially, I was hoping to write test code in F#, for readability by testers. – Safford

There were also good suggestions here: #1989987 – Timekeeper

Answer 1

My own unit testing library, Unquote, takes advantage of F# quotations to allow you to write test assertions as plain, statically checked F# boolean expressions and automatically produces nice step-by-step test failure messages. For example, the following failing xUnit test

[<Fact>]
let ``demo Unquote xUnit support`` () =
    test <@ ([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0] @>

produces the following failure message

Test 'Module.demo Unquote xUnit support' failed: 

([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0]
[4; 3; 2; 1] = [4..1]
[4; 3; 2; 1] = []
false

        C:\File.fs(28,0): at Module.demo Unquote xUnit support()

FsUnit and Unquote have similar missions: to allow you to write tests in an idiomatic way, and to produce informative failure messages. But FsUnit is really just a small wrapper around NUnit Constraints, creating a DSL which hides object construction behind composable function calls. But it comes at a cost: you lose static type checking in your assertions. For example, the following is valid in FsUnit

[<Test>]
let test1 () =
    1 |> should not (equal "2")

But with Unquote, you get all of F#'s static type-checking features so the equivalent assertion would not even compile, preventing us from introducing a bug in our test code

[<Test>] //yes, Unquote supports both xUnit and NUnit automatically
let test2 () =
    test <@ 1 <> "2" @> //simple assertions may be written more concisely, e.g. 1 <>! "2"
    //           ^^^
    //Error 22 This expression was expected to have type int but here has type string

Also, since quotations are able to capture more information at compile time about an assertion expression, failure messages are a lot richer too. For example the failing FsUnit assertion 1 |> should not (equal 1) produces the message

Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed: 
  Expected: not 1
  But was:  1
    C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\FsUnit.fs(11,0): at FsUnit.should[a,a](FSharpFunc`2 f, a x, Object y)
    C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()

Whereas the failing Unquote assertion 1 <>! 1 produces the following failure message (notice the cleaner stack trace too)

Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed: 

1 <> 1
false

    C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()

And of course from my first example at the beginning of this answer, you can see just how rich and complex Unquote expressions and failure messages can get.

Another major benefit of using plain F# expressions as test assertions over the FsUnit DSL, is that it fits very well with the F# process of developing unit tests. I think a lot of F# developers start by developing and testing code with the assistance of FSI. Hence, it is very easy to go from ad-hoc FSI tests to formal tests. In fact, in addition to special support for xUnit and NUnit (though any exception-based unit testing framework is supported as well), all Unquote operators work within FSI sessions too.

Farrish

I tried Unquote because of all the upvotes here, and after a week I am sold on it. So far I've found valuable: writing tests in fsi, then moving them to NUnit; the integration with NUnit; the strong typing of test expressions (Unquote found a test I had which was only passing by chance); the quotation utilities, such as "eval". If you have Nuget, then Unquote can be installed with "install-package Unquote". – Tendril

Answer 2

I haven't yet tried Unquote, but I feel I have to mention FsCheck: http://fscheck.codeplex.com/ This is a port of Haskells QuickCheck library, where rather than specifying what specific tests to carry out, you specify what properties about your function should hold true. To me, this is a bit harder than using traditional tests, but once you figure out the properties, you'll have more solid tests. Do read the introduction: http://fscheck.codeplex.com/wikipage?title=QuickStart&referringTitle=Home

I'd guess a mix of FsCheck and Unquote would be ideal.

Atingle

I've not tried it myself yet, but I've heard that FsCheck and Unquote mix very well together indeed. – Farrish

Answer 3

You could try my unit testing library Expecto; it's has some features you might like:

• F# syntax throughout, tests as values; write plain F# to generate tests
• Use the built-in Expect module, or an external lib like Unquote for assertions
• Parallel tests by default
• Test your Hopac code or your Async code; Expecto is async throughout
• Pluggable logging and metrics via Logary Facade; easily write adapters for build systems, or use the timing mechanism for building an InfluxDB+Grafana dashboard of your tests' execution times
• Built in support for BenchmarkDotNet
• Build in support for FsCheck; makes it easy to build tests with generated/random data or building invariant-models of your object's/actor's state space

Hello world looks like this

open Expecto

let tests =
  test "A simple test" {
    let subject = "Hello World"
    Expect.equal subject "Hello World" "The strings should equal"
  }

[<EntryPoint>]
let main args =
  runTestsWithArgs defaultConfig args tests

Seedman