What is the best practice when writing Perl tests that involve randomness?

Question

While working on some updates to my module List::Gen, I decided to add a ->pick(num) method, which will return a num sized list of random elements from its source. To test this, I used srand to seed the random number generator, and made several tests of the form:

srand 1234;
is $src->pick(5)->str, '3 6 1 7 9';

And this all worked well on the Windows machine I was on at the time. However, when I moved the project over to a Mac workstation, all of the randomness tests failed, since despite having the same random seed, rand was producing different results. I gather this is from different underlying C implementations of rand().

So the question is, what is the best cross platform way to test these functions? Should I overload the rand function with my own? Should I build in hooks to the functions that use rand to enable a "testing" mode that produces predicable output? Are there other methods?

I would prefer answers that include core Perl techniques, as I am trying to keep the module's dependency tree small.

Test::Random and Test::MockRandom seem to be CPAN's suggestions, does anyone have experience with these modules?

Susan

Test::MockRandom allows you to pick the sequence of numbers so that you'll be immune to system differences. It's the way I've gone the one time I needed random output and wanted to unit test it. – Bertrando

Answer 1

I have not used either one.

Looks like Test::Random would be a better choice for you since you are apparently just using randomness in your testing, not in your released code. It should be a lot simpler to use.

The Test::MockRandom module forces the rand() function to return a deterministic sequence.

Rachaba

Answer 2

You could run a few picks and make sure they don't all return the same thing. That's the purpose of the function, after all.

Deafanddumb

Answer 3

I prefer to simply encapsulate the environmental dependency and override it for testing purposes, a test pattern called Test Stub. Test Stub also covers other indirect inputs like the system time and file handles. These should all be construed as different forms of the same problem, which is why I think the CPAN solutions to this problem less-than-great.

Applied to the random number domain, we have something like:

use strict;
use warnings;

package Foo;

sub new {
    my ($class) = @_;

    return bless {} => $class;
}

sub get_random_number {
    return rand();
}

package main;
use Test::MockObject::Extends;
use Test::More tests => 1;

my $foo = Test::MockObject::Extends->new( Foo->new() );
$foo->set_series(get_random_number => 0.5, 0.001, 0.999);

is( $foo->get_random_number, 0.5 );

This leaves the system under test unchanged except for a refactoring it should have anyway, but provides control points to inject predictable data into the test. get_random_number will not be covered by tests, so it is vital that it is written in such a way as to be correct on inspection; a single call to the depended-upon system resource is about all that should be in there.

In the case of your specific problem, you need to factor the dependency on rand out of pick, then override the extracted method in a testable version of List::Gen. Test::MockObject::Extends is pretty much ideally suited to this need.

Musketeer

Answer 4

Maybe the random part does not matter for your tests?

The passing tests could check for the following:

1. did ->pick( X ) return X elements?
2. Are all X elements part of the $src list?
3. Test for 0, 1, etc...
4. (maybe?) Test that two different srand seeds return different lists

This is essentially what you are already doing, since you are trying to take rand() out of the equation. May as well go all the way and test that your function does what it says on the tin.

Oogonium