How to manually create TAP output

Question

There is a great Perl module Test::More that everybody uses for unit testing. Here is the very simple script t/sample_1.t:

use Test::More tests => 1;

fail('This test fails');

I wanted to write script that does the same thing, but without Test::More.

I've read several the docs about TAP (test anything protocol) to find out how to write the script. I've read:

• Wikipedia article about TAP
• TAP specification

Unfortunately the documentation wasn't enough. I had to examine the output of script that uses Test::More to find out that I need to output diagnostics to STDERR (there was nothing about this in the docs).

So, I have written a script that does completely the same things as the script with Test::More script. Here is the listing of t/sample_2.t:

$| = 1;

print "1..1\n";
print "not ok 1 - This test fails\n";
print STDERR "#   Failed test 'This test fails'\n";
print STDERR "#   at t/sample_1.t line 3.\n";
print STDERR "# Looks like you failed 1 test of 1.\n";

exit 1;

But when using prove these 2 scripts output different things. The line "# Failed test 'This test fails'" in prove is displayed on different lines for different tests. Here is the screenshot:

I've written a test scripts that uses Capture::Tiny to check that STDERR, STDOUT and exit code for both scripts a identical. And the script shows that both scripts output the same things.

I've stored all the test files and a test script at GitHub repo.

My question. How should I write Perl unit test without Test::More to have the same output as with Test::More.

PS If you are interested why I need this. I need this to solve the issue of my Perl module Test::Whitespaces.

Guttery

Why not fix your module to use Test::Builder properly? Then you can send diagnostic output (which appears with or without prove -v) using diag. – Apples

Thank you for the idea! I didn't know about Test::Builder. Test::Builder is in Perl core. I have created a sample script that does exactly what script t/sample_1.t does: github.com/bessarabov/manual_tap_generation/blob/master/t/… I will try to use it in my module. But I still don't understand why I can't create the output without any module. – Guttery

Answer 1

At last I have found out what is going on.

hobbs has advised me to use Test::Builder. I created test script with Test::Builder that worked exaclty as the script with Test::More (here it is).

Then I started examinig source code of Test::Builder to find out why the source of such behaviour. Here is the part of lib/TB2/Formatter/TAP/Base.pm file:

# Emit old style comment failure diagnostics
sub _comment_diagnostics {
    my($self, $result) = @_;

    ...

    # Start on a new line if we're being output by Test::Harness.
    # Makes it easier to read
    $self->$out_method("\n") if ($out_method eq 'err') and $ENV{HARNESS_ACTIVE};
    $self->$diag_method($msg);

    return;
}

So, this is the answer. prove sets up special environment variable HARNESS_ACTIVE and Test::More and friends puts additional line break symbol "\n" before any diagnostics that are printed to STDERR.

At last I've created test script that outputs exactly the same as the script written with Test::More. Source code of the script.

I really don't like this solution. It took me and outher peopler much time to find out what is going on. I'm sure that the task of pretty output should be solved in TAP parsers, and not in TAP producers.

=(

Guttery

but TAP parsers job is to parse TAP, and the diagnostic stream (stderr) is not TAP. – Sheetfed

Oh. This makes sense. I haven't realized that STDERR output is not part of TAP protocol. Hm. I think I's a pity that TAP protocol has no specification of how to output diagnostic data to the user. Because there is no such thing in the protocol the prove has to invent some hack that will make users life better. – Guttery

TAP does cover diagnostics - any lines beginning with a hash symbol are treated as diagnostics. – Sodom

Answer 2

While I've got absolutely no frickin idea what's going on, I can get the outputs to match (visually at least) by including the following before any other output to STDERR:

print STDERR "\r";

This makes them match visually when run through prove or plain old perl. However, this is NOT what Test::More is doing.

The TAP you're outputting is per spec; if prove wants to treat it differently from the TAP Test::More is outputting, I'd argue that's a bug (or at least an oddity) in prove. Personally when I've written Test modules, I've always used Test::Builder or wrapped Test::More to output the TAP. Each of these is a core module. This seems to be what the majority of Test modules tend to do.

Sodom

Than you! You were absolutely right. I should have written line break before any output to STDERR. It's strange but this is what Test::More is doing in case environment variable HARNESS_ACTIVE is true. – Guttery