Run PHPUnit Tests in Certain Order

S

9

82

Is there a way to get the tests inside of a TestCase to run in a certain order? For example, I want to separate the life cycle of an object from creation to use to destruction but I need to make sure that the object is set up first before I run the other tests.

Sherborn answered 13/8, 2008 at 19:2 Comment(2)

You can add @depends as described in an answer below, and using setup() and teardown() is also a good idea, but tests are just run top to bottom... – Hemlock 10/6, 2015 at 15:30

One additional use-case that doesn't seem to have been covered: Maybe all tests are atomic, but some tests are SLOW. I want the fast tests to be run ASAP so they can fail fast, and any slow tests to be run dead last, after I've already seen other problems and can get to them immediately. – Achromatize 19/3, 2016 at 21:31

E

63

Maybe there is a design problem in your tests.

Usually each test must not depend on any other tests, so they can run in any order.

Each test needs to instantiate and destroy everything it needs to run, that would be the perfect approach, you should never share objects and states between tests.

Can you be more specific about why you need the same object for N tests?

Elaelaborate answered 13/8, 2008 at 20:13 Comment(15)

This doesn't seem correct to me. The point of a unit test is to test an entire unit. The point of having a unit is to group things together that have to depend on each other. Writing tests that test individual methods without the context for the class is akin to advocating procedural programming over oo because you're advocating that individual functions should not depend on the same data. – Heaven 4/5, 2013 at 20:35

I disagree with your point of view. The output of an instantiation test is a valid object that can be used by other tests in your test suite. There's no need to instantiate a new object for each test, specially if the constructor is complicated. – Engineer 12/7, 2013 at 15:8

If the constructor is complicated you are doing something wrong, probably your class is doing too much. Please read about "SOLID", more specific about the "Single Responsibility Pattern (SRP)", also you should "fake" the dependencies in your tests using mocks, please read about "mocks, fakes and stubs" too. – Elaelaborate 13/7, 2013 at 22:15

There may also be a practical reason. For instance, if the cleanup you need to do is particualrly time consuming, you might use the tearDownAfterClass function so as to only run it once. If one particular test requires a clean slate, then you either have to make sure that test gets run first, or manually call the tearDownAfterClass function at its start, causing it to be run twice. Yes, this is probably a sign that something is wrong with the test class design, but there are legitimate cases where ordering tests is useful. – Statesmanship 23/7, 2014 at 13:54

At least for database testing reusing objects (at least the connection) is often necessary. PHPUnit refers to that as well: phpunit.de/manual/current/en/database.html (see: Tip: Use your own Abstract Database TestCase) – Engird 28/7, 2017 at 8:12

@Engird if you are testing againt a real database, you are not doing unit tests. You are doing functional tests. To do unit tests, you must mock the DB adapter and, as Fabio says, you need to instantiate your SUT (System Under Test) at each test-run. You can use the protected setUp() method to prepare the mocks if there is stuff you are going to repeat for each test. – Predestinate 31/3, 2018 at 14:25

I don't agree - I am testing a process where a "widget" must be submitted by one person and approved by another, so they must be in order. – Puncheon 21/8, 2018 at 1:30

@Xavi Montero: When actually developing a querybuilder and/or an ORM, you need to execute queries on the corresponding DBMS for a full test. in this specific case, i think this counts as unit testing. – Engird 15/2, 2019 at 10:8

@Engird if the code you are testing is actually the bridge to the DB (for example, the ORM, ODM, etc) (for example you are a developer at the Doctrine project or your company is developing a Doctrine's substitute) you'd probably be mocking the connection object and testing how the connection is consumed to see what queries are created etc. The only test that could have an entry to a real-life DB could be the test for the Connection object itself, and it would access a very controlled real-connection with limited tests. – Predestinate 16/2, 2019 at 2:28

@Engird but if you refer to testing classes that generally access an ORM you "should" mock the ORM itself. Instead, it's often difficult to mock the ORM and for "simplicity" we want to allow the ORM to be "real" (not mocked) and go for a real INSERT or SELECT. [continues in next comment] – Predestinate 16/2, 2019 at 2:37

Okey, I agree with you: For some specific cases, doing a real access to a DB could count as a "unit" test, not "funcional" test with the following considerations: [continues in next comment] – Predestinate 16/2, 2019 at 2:40

a) We know we are breaking the "unit" test, as it tests the consuming_class+ORM instead of the consuming_class only. We will allow this if and only if the ORM is fully tested. Testing one and only class is nearer to "unit testing" and testing two or more classes is nearer to integration testing. – Predestinate 16/2, 2019 at 2:40

b) If we extend the DBUnit and use for instance the datasets utilities to create sets of data and preferrably use in-memory sqlite, is nearer to "unit-testing", but if we connect to a mysql test-database with a copy of the production data it's nearer functional testing – Predestinate 16/2, 2019 at 2:40

c) But for me, finally, the content of the test is actually the one that marks if we are in "unit" or "functional". If we test a single operation; for example that an e-mail adapter logs an event in a database saying "email sent" just after returning from the SMTP call; and say we mock the SMTP but do not mock the DB for simplicity; this could be a "unit test" because we test one single method on a single class EmailAdapter->sendEmail() and check how this altered the state. [continues] – Predestinate 16/2, 2019 at 2:44

Instead, if we test for example that after saving an object we can query it, that is NOT unit but functional. We are testing something like $adapter->save( $object ); and later $actualObject = $adapter->loadById( $id ); - That tests the combination of 2 methods and this is functional testing, not unit testing. If we want to specifically adhere to unit, this should be splitted in 2 tests: One for each unit: test 1) with a setup cleaning the DB, persisting and asserting the DB contains data, test 2) setup sets a fixture, load and assert the loaded object matches the fixture. – Predestinate 16/2, 2019 at 2:53

T

161

PHPUnit supports test dependencies via the @depends annotation.

Here is an example from the documentation where tests will be run in an order that satisfies dependencies, with each dependent test passing an argument to the next:

class StackTest extends PHPUnit_Framework_TestCase
{
    public function testEmpty()
    {
        $stack = array();
        $this->assertEmpty($stack);

        return $stack;
    }

    /**
     * @depends testEmpty
     */
    public function testPush(array $stack)
    {
        array_push($stack, 'foo');
        $this->assertEquals('foo', $stack[count($stack)-1]);
        $this->assertNotEmpty($stack);

        return $stack;
    }

    /**
     * @depends testPush
     */
    public function testPop(array $stack)
    {
        $this->assertEquals('foo', array_pop($stack));
        $this->assertEmpty($stack);
    }
}

However, it's important to note that tests with unresolved dependencies will not be executed (desirable, as this brings attention quickly to the failing test). So, it's important to pay close attention when using dependencies.

Tabina answered 16/12, 2009 at 14:46 Comment(3)

For PHPUnit, this means the test function will be skipped if the previous test weren't executed. That does not create a test order. – Revers 9/2, 2014 at 19:12

Just to expand on @Dereckson, the @depends annotation will cause a test to be skipped if the test that is dependent on either hasn't been run yet or failed when it did run. – Gati 21/8, 2014 at 0:39

@Gati does that mean when we put or write testPop() method before testPush() in the file then testPop() would never be executed and get always skipped? – Rajput 28/3, 2019 at 6:34