Organizing Haskell Tests

Question

So I'm trying to follow the suggested structure of a Haskell project, and I'm having a couple problems organizing my tests.

For simplicity, let's start with:

src/Clue/Cards.hs # defines Clue.Cards module
testsuite/tests/Clue/Cards.hs # tests Clue.Cards module

For one, I'm not sure what to name the module in testsuite/tests/Clue/Cards.hs that contains the test code, and for another, I'm no sure how to compile my test code so that I can link to my source:

% ghc -c testsuite/tests/Clue/Cards.hs -L src
testsuite/tests/Clue/Cards.hs:5:0:
    Failed to load interface for `Clue.Cards':
      Use -v to see a list of the files searched for.

Overweening

Answer 1

I use myself the approach taken by Snap Framework for their test-suites, which basically boils down to:

1. Use a test-framework such as haskell-test-framework or HTF

2. Name the modules containing tests by appending .Tests to the module-name containing the IUT, e.g.:

   module Clue.Cards where ... -- module containing IUT
   
   module Clue.Cards.Tests where ... -- module containing tests for IUT
   

3. By using separate namespaces, you can put your tests in a separate source-folder tests/, you can then use a separate Cabal build-target (see also cabal test-build-target support in recent Cabal versions) for the test-suite which includes the additional source folder in its hs-source-dirs setting, e.g.:

   Executable clue
     hs-source-dirs: src
     ...
   
   Executable clue-testsuite
     hs-source-dirs: src tests
     ...
   

   This works, since there's no namespace collision between the modules in your IUT and the test-suite anymore.

Germ

+1 for mentioning snap-framework, which is extremely well-organized in this regard. – Piddling

Cool. I'm using this project as a way to learn the Haskell ecosystem (I don't think anyone's itching for an implementation of the rules of Clue/Cluedo), and I haven't tackled cabal yet, so this is a good kick in the pants. I'll figure out how to use cabal, then circle back around to testing. – Overweening

maybe also noteworthy about the snap-framework project: they have their test-suites integrated with hudson, publishing the test-results and coverage reports (see buildbot.snapframework.com) – Germ

Is there any downside of naming your modules Tests.Clue.Cards instead of Clue.Cards.Tests? – Cub

Sorry for the newbie question, but if you name your module Clue.Cards.Tests does that mean your test file is called something like tests/Clue.Cards.Tests.hs? Or would it be tests/Clue/Cards/Tests.hs? – Postpositive

@Postpositive - I am admittedly new to Haskell testing, but I had to follow the tests/Clue/Cards/Tests.hs file structure in order for my tests to be found/compiled. – Verlaverlee

Answer 2

Personally I feel that an extra ./src/ directory doesn't make much sense for small Haskell projects. Of coarse there's source, I downloaded the source code.

Either way (with or without src), I'd suggest you refactor and have a Clue directory and a Test directory:

./Clue/Cards.hs   -- module Clue.Cards where ...
./Test/Cards.hs   -- module Test.Cards where ...

This allows GHCi + Test.Cards to see Clue.Cards without any extra args or using cabal. On that note, if you don't use cabal + flags for optionally building your test modules then you should look into it.

Another option, which I use in many of my projects, is to have:

./Some/Module/Hierarchy/File.hs
./tests/someTests.hs

And I cabal install the package then run the tests/someTests.hs stuff. I guess this would be annoying if my packages were particularly large and too a long time to install.

Saiz

I think an additional src directory always makes sense because it qualifies what's actually contained within the source-code hierarchy: Haskell sources! This is especially useful when your application has other things lying about in it, such as scripts, config files, and possibly other non-Haskell sources. – Breathless

Answer 3

Here's another way:

Each module's unit tests are defined as a hunit TestList at the end of the module, with some consistent naming scheme, such as "tests_Path_To_Module". I think this helps me write tests, since I don't have to search for another module far away in the source tree, nor keep two parallel file hierarchies in sync.

A module's test list also includes the tests of any sub-modules. Hunit's runTestTT runner is built in to the app, and accessible via a test command. This means a user can run the tests at any time without special setup. Or if you don't like shipping tests in the production app, use CPP and cabal flags to include them only in dev builds, or in a separate test runner executable.

There are also functional tests, one or more per file in the tests/ directory, run with shelltestrunner, and some dev-process-related tests based in the Makefile.

Batholith

I don't recommend putting tests in your implementation modules. You should be able to compile your production modules without a dependency on the test framework. (xUnit Patterns calls this smell "Test Dependency In Production"). Using CPP flags to extricate them is not a real solution. In any case, it breaks the Single Responsibility Principle. – Louvain

Answer 4

For completeness sake, it worth mentioning a very easy approach for small project through ghci -i. For example, in your case,

>ghci -isrc:testsuite
ghci>:l Clue.Cards
ghci>:l tests.Clue.Cards

Dorwin