What is the preferred unit testing framework for Perl?

Question

I'm sort of new to Perl and I'm wondering if there a prefered unit testing framework?

Google is showing me some nice results, but since I'm new to this, I don't know if there is a clear preference within the community.

Lint

Answer 1

Perl has a MASSIVE set of great testing tools that come with it! The Perl core has several tens of thousands of automated checks for it, and for the most part they all use use these standard Perl frameworks. They're all tied together using TAP - the Test Anything Protocol.

The standard way of creating TAP tests in Perl is using the Test::More family of packages, including Test::Simple for getting started. Here's a quick example:

use 5.012;
use warnings;

use Test::More tests => 3;

my $foo = 5;
my $bar = 6;

ok $foo == 5, 'Foo was assigned 5.';
ok $bar == 6, 'Bar was assigned 6.';
ok $foo + $bar == 11, 'Addition works correctly.';

And the output would be:

ok 1 - Foo was assigned 5.
ok 2 - Bar was assigned 6.
ok 3 - Addition works correctly.

Essentially, to get started, all you need to do is put pass a boolean value and a string explaining what should occur!

Once you get past that step, Test::More has a large number of other functions to make testing other things easier (string, regex compares, deep structure compares) and there's the Test::Harness back end that will let you test large groups of individual test scripts together.

On top of that, as Schwern pointed out, almost all of the modern Test:: modules work together. That means you can use Test::Class (as pointed out by Markus) with all of the great modules listed in rjh's answer. In fact, because Test::Builder--the tool that Test::More and others are built on (and currently maintained by Schwern...thanks Schwern!)--you can, if needed, build your OWN test subroutines from the ground up that will work with all the other test frameworks. That alone makes Perl's TAP system one of the nicest out there in my opinion: everything works together, everyone uses the same tool, and you can add on to the framework to suit your needs with very little additional work.

Graehme

just wondering, what is tests => 3. do i need to hardcode the number of cases? – Lint

You don't have to. But if your test script dies after outputting results for only 1 or 2 tests, Test::More knows that your test has failed (even if it exits with an error code of zero). You can also do use Test::More 'no_plan' or call done_testing() at the end of your script. – Nutrient

I can't find the proper way of using no plan. If I don't specify a number argument to done_testing I get You tried to run a test without a plan! Gotta have a plan – Lint

What version of Perl and Test::More are you using? perl -v will give you the perl version, and you can get Test::more from perl -MTest::More -e "print Test::More->VERSION()" Either way, it's almost always better to have a plan, if you can. I found it was tedious at first, but a lifesaver later when the test was unexpectedly dying halfway through, and not reporting several of my test cases. If it wasn't for the plan, I would've never known there was a problem. – Graehme

i'm still new to perl, so i don't understand all the pecularities, but it seems like poor practice to require anything to be hard-coded. i've used frameworks in java,c++, python, etc and each has had a means of coping with a crashing test. i'm just thinking there has to be some evasion that will still preserve full testing functionality. thanks for your answer robert. – Lint

The "preferred" framework wants a test plan. I, personally, just run the test and verify that the whole test was completed. When the test is done, I have the number of tests. I just use that number to set the plan from the info in the results. – Hoicks

If you are willing to use a complex, new, cutting edge testing module, you might want to look at Test::Sweet -- search.cpan.org/dist/Test-Sweet/lib/Test/Sweet.pm It does look pretty sweet, but I have yet to try it in production. – Hoicks

On newer Test::More (>=0.88), use Test::More; followed by done_testing; at the end should work fine. Otherwise, you need to do use Test::More qw(no_plan). – Nutrient

@Robert P: Test::Harness should detect and complain about non-zero exit codes, so unless your script called exit(0), it should have noticed. – Nutrient

@Mike: it's just another failsafe that's available; it's not required. Ideally you already know how many tests ran. In fact, Test::More will tell you. For example, if you add another test to the script without changing the tests declaration at the top, it will say something like # Looks like you planned 3 tests but ran 4., making it a simple matter to change it. I'd also argue that the whole point of a unit test is to 'hard code' values to try. If your unit tests run variable amounts of steps every time, it's not very repeatable now, is it? :) – Graehme

Thanks Robert, you've been very helpful +1 – Lint

Its worth noting that just about all the modern Test:: modules work together. Test::More can be combined with Test::Differences and Test::Deep and so on. – Athene

@Schwern: Absolutely - that's so important I think I'll add it to the answer. – Graehme

Answer 2

Perl's most popular test 'framework' is a test results format known as TAP (Test Anything Protocol) which is a set of strings that look like:

ok 1 - Imported correctly
ok 2 - foo() takes two arguments
not ok 3 - foo() throws an error if passed no arguments

Any script that can generate these strings counts as a Perl test. You can use Test::More to generate TAP for various conditions - checking if a variable is equal to a value, checking if a module imported correctly, or if two structures (arrays/hashes) are identical. But in true Perl spirit, there's more than one way to do it, and there are other approaches (e.g. Test::Class, which looks a bit like JUnit!)

A simple example of a test script (they usually end in .t, e.g. foo.t)

use strict;
use warnings;
use Test::More tests => 3;  # Tell Test::More you intend to do 3 tests

my $foo = 3;
ok(defined $foo, 'foo is defined');
is($foo, 3, 'foo is 3');
$foo++;
is($foo, 4, 'incremented foo');

You can use Test::Harness (commonly invoked as prove from the shell) to run a series of tests in sequence, and get a summary of which ones passed or failed.

Test::More can also do some more complex stuff, like mark tests as TODO (don't expect them to pass, but run them just in case) or SKIP (these tests are broken/optional, don't run them). You can declare the number of tests you expect to run, so if your test script dies half-way, this can be detected.

Once you begin to do more complex testing, you might find some other CPAN modules useful - here are a few examples, but there are many (many) more:

Test::Exception - test that your code throws an error/doesn't throw any errors
Test::Warn - test that your code does/doesn't generate warnings
Test::Deep - deeply compare objects. They don't have to be identical - you can ignore array ordering, use regexes, ignore classes of objects etc.
Test::Pod - make sure your script has POD (documentation), and that it is valid
Test::Pod::Coverage - make sure that your POD documents all the methods/functions in your modules
Test::DBUnit - test database interactions
Test::MockObject - make pretend objects to control the environment of your tests

Nutrient

Answer 3

Definitely start with this page: http://perldoc.perl.org/Test/Simple.html and follow the reference to Test::Tutorial.

Instrumentalism

Answer 4

If you practice TDD, you will notice that your set of unit tests are changing A LOT. Test::Class follows the xUnit patterns (http://en.wikipedia.org/wiki/XUnit).

For me, the main benefit with xUnit is the encapsulation of each test in methods. The framework names each assertion by the name of the test method, and adds the possibility to run setup- and teardown methods before and after each test.

I have tried the "perl-ish" way for unit testing also (just using Test::More), but I find it kind of old-fashioned and cumbersome.

Barcelona

I really like Test::Class too. The neat thing about Test::Class is that it's simply another way of building TAP based tests - another framework, converting it from "Procedural" to "Object oriented". It separates out the "how its reported" from "How its run", letting you use any of the other Test::More style test steps / test functions within your Test::Class object. BUT it doesn't have to be for TAP based tests either - it just happens to be very good at it. You can extend Test::Class and override many of the default behaviors, and simply use the framework. – Graehme

Answer 5

Some anti-recommendations may be in order:

Anti-recommendation:

Do NOT use the Test::Unit family of test packages for Perl, such as Test::Unit::Assert and Test::Unit::TestCases.

Reason: Test::Unit appears to be abandoned.

Test::Unit, Test::Unit::TestCases, Test::Unit::Assert work, pretty well (when I used them 2015-2016). Test::Unit is supposedly not integrated with Perl's Test Anything Protocol (TAP), although I found that easy to fix.

But Test::Unit is frustrating because so many of the other Perl test packages, mostly built using Test::Builder, like Test::More, Test::Most, Test::Exception, Test::Differences, Test::Deep, Test::Warn, etc., do NOT interact well with the object oriented testing approach of Test::Unit.

You can mix Test::Unit tests and Test::Builder tests once you have adapted Test::Unit to work with Test::More and the TAP; but the good features of these other packages are not available for OO extension. Which is much of the reason to use an xUnit-style test anyway.

Supposedly CPAN's Test::Class allows to "Easily create test classes in an xUnit/JUnit style" -- but I am not sure that I can recommend this. It certainly doesn't look like xUnit to me - not OO, idiosyncratic names like is(VAL1,VAL2,TESTNAME) instead of xUnit style names like $test_object->assert_equals(VAL1,VAL2,TEST_ERR_MSG). Test::Class does have the pleasant feature of auto-detecting all tests annotated :Test, comparable to xUnit and TEST::Unit::TestCase's approach of using introspection to run all functions named test_*.

However, the underlying package Test::Builder is object oriented, and hence much more xUnit style. Don't be scared away by the name - it's not a factory, it's mostly a suite with test assert methods. Although most people inherit from it, you can call it directly if you wish, e.g. $test_object->is(VAL1,VAL2,TESTNAME), and often you can use Test::Builder calls to work around the limitations of procedural packages like Test::More that are built on top of Test::Builder - like fixing the callstack level at which an error is reported.

Test::Builder is usually used singleton style, but you can create multiple objects. I am unsure as to whether these behave as one would expect from an xUnit family test.

So far, no easy way to work around limitations such as Perl TAP tests use TEST_NAMES, per assert, without hierarchy, and without distinguishing TEST_NAMES from TEST_ERROR_MESSAGES. (Err reporting level helps with that lack.)

It may be possible to create an adapter that makes Test::Builder and TAP style tests more object oriented, so that you can rebase on something other than TAP (that records more useful info than TAP - supposedly like ANT's XML protocol). I think to adapt the names and/or the missing concepts will either involve going into Test::Builder, or introspection.

Honeysuckle

Thanks, Peter. Much cleaner. I start frothing when I have wasted time using aging software. – Honeysuckle

Answer 6

Test2::V0 is the sequel to Test::More...

...and the yath command from Test2::Harness, the sequel to prove.

There's a degree of backwards compatibility, so worth a look.

https://metacpan.org/pod/Test2::Manual

Desperate