What do you need from a test harness?

Question

I'm one of the people involved in the Test Anything Protocol (TAP) IETF group (if interested, feel free to join the mailing list). Many programming languages are starting to adopt TAP as their primary testing protocol and they want more from it than what we currently offer. As a result, we'd like to get feedback from people who have a background in xUnit, TestNG or any other testing framework/methodology.

Basically, aside from a simple pass/fail, what information do you need from a test harness? Just to give you some examples:

• Filename and line number (if applicable)
• Start and end time
• Diagnostic output such as the difference between what you got and what you expected.

And so on ...

Ninon

Answer 1

Most definitely all things from your list for each individual item:

• Filename
• Line number
• Namespace/class/function name
• Test coverage
• Start time and end time
• And/or total time (this would be more useful for me than the top two items)
• Diagnostic output such as the difference between what you got and what you expected.

From the top of my head not much else but for the group of tests I would like to know

• group name
• total execution time

Corny

Answer 2

It must be very, very easy to write a test, and equally easy to run them. That, to me, is the single most important feature of a testing harness. If someone has to fire up a GUI or jump through a bunch of hoops to write a test, they won't use it.

Debouch

It looks like TAP doesn't actually address writing tests at all, just how one might communicate test results back to some other system (for reporting purposes or somesuch) – Danadanae

Answer 3

An arbitrary set of tags - so I can mark a test as, for example "integration, UI, admin".

(you knew I was going to ask for this didn't you :-)

Ranitta

Hey, you can ask for this if I can ask for a bunch of features in Test::Class :) – Ninon

Yes, I want this sort of thing in the harness. I was talking to Schwern about this at the last Portland Linux Users Group meeting. I want something like Test::Manifest, but for individual tests. :) – Damp

Answer 4

To what you said I'd add:

• Method/function/class name
• Coverage counting tool, with exceptions (Do not count these methods)
• Result of N last runs available
• Mandate that ways to easily parse test results must exist

Zea

Answer 5

Any sort of diagnostic output - especially on failure is critical. If a test fails, you don't want to always have to rerun the test under a debugger to see what happened - there should be some cludes in the output.

I also like to see a before and after snapshot of critical system variables like memory or hard disk space available as those can provide great clues as well.

Finally, if you're using random seeds for any of the tests, write the seed out to the logfile so that the test can be reproduced if necessary.

Alimony

None of that can be done in the framework (it doesn't know what your test does, only if it succeeds), but TAP gives you the ability to add this information when you write your tests with the diag function. – Treenware

Answer 6

I'd like the ability to concatenate and nest TAP streams.

Serow

I know that this has been discussed. I'm just impatient. :D – Serow

Answer 7

A unique id (uuid, md5sum) to be able to identify an individual test -- say, for use when inserting test results in a database, or identifying them in a bug tracker to make it possible for QA to rerun an individual test.

This would also make it possible to trace an individual test's behavior from build-to-build through the entire lifecycle of multiple revisions of a product. This could eventually allow larger-scale correlations between 'historic' events (new hire, product release, hardware upgrades) and the profile(s) of tests that fail as a result of such events.

I'm also thinking that TAP should be emitted through a dedicated side-channel rather than mixed in with stdout. I'm not sure this is under the scope of the protocol definition.

Trimurti

Answer 8

I use TAP as output protocol for a set of simple C++ test methods, and have seen the following shortcomings:

• test steps cannot be put into groups (there's only the grouping into several test scripts; but for running all tests in our software, I need at least one more level of grouping, so that a single test step would be identified by like "DB connection" -> "Reconnection Test" -> "test step #3")
• seeing differences between expected and actual output is useful; I either print the diff to stderr (as comment) or actually launch a graphical diff tool
• the protocol and tools must be really language-independent. For example, so far I only know of the Perl "prove" tool for running tests, which is limited to running Perl scripts

In the end, the test output must be suitable as basis for easily generating an HTML report file which lists succeeded tests very concisely, gives detailed output for failed tests, and makes it possible to quickly jump into the IDE to the failing test line.

Sideband

Answer 9

• optional ascii coloured output, green for good, yellow for pending, red for errors

• the idea of things being pending

• a summary at the end of the test report of commands that will run the individual tests where

• List item

  • something went wrong

  • something in the test was pending

Backus

What do you mean by "the idea of things being pending"? – Fractional

Answer 10

Extension idea for TAP:

1..4
ok 1 - yay
not ok 2 - boo
ok 3 - yay #json:{...}
ok 4 - see my json

Ability to attach a #json comment... - can be safely ignored by existing code - well-defined tags can be easily reserved at testanything.org - easy to produce, parse and read complex types - yaml is a pain

Volatilize