HUnit/QuickCheck with Continuous Integration

Question

Are there any extensions to HUnit or QuickCheck that allow a continuous integration system like Bamboo to do detailed reporting of test results?

So far, my best idea is to simply trigger the tests as part of a build script, and rely on the tests to fail with a non-zero exit code. This is effective for getting attention when a test fails, but confuses build failures with test failures and requires wading through console output to determine the problem's source.

If this is the best option with current tools, my thought is to write a reporting module for HUnit that would produce output in the JUnit XML format, then point the CI tool at it as though it were reporting on a Java project. This seems somewhat hackish, though, so I'd appreciate your thoughts both on existing options and directions for new development.

Kenosis

Mimicking JUnit XML format seems like a pretty decent way to get the most bang for your coding buck. See if HUnit already has support for it though. QuickCheck is a bit trickier, because it's not really traditional XUnit style, though you could probably shoehorn it as well. – Cadence

maybe slightly related: https://mcmap.net/q/427907/-organizing-haskell-tests – Cerargyrite

Answer 1

The test-framework package provides tools for integrating tests using different testing paradigms, including HUnit and QuickCheck, and its console test runner can be passed a flag that makes it produce JUnit-compatible XML. We use it with Jenkins for continuous integration.

Invocation example:

$ ./test --jxml=test-results.xml

Azote

Btw, did you manage to integrate code-coverage graphing as well in Jenkins? – Cerargyrite

Thanks for this! I agree with @Cerargyrite that it would be very cool to integrate hpc reports as well. It shouldn't be too hard with hpc markup... – Kenosis

How do I get the --jxml into a cabal install --enable-tests-invocation? – Currin

Answer 2

I've just released a package which generates test-suites based off modules containing quickCheck properties: http://hackage.haskell.org/package/tasty-integrate

This is one step above test-framework/tasty at the moment, as it forcefully pulls/aggregates them off the filesystem, instead of relying upon per-file record keeping. I hope this helps your CI process.

Stank