How can I speed up my PHPUnit + DBUnit test suite execution?
Asked Answered
F

2

11

I'm running in to some real speed issues with PHPUnit/DBUnit. Anything that extends PHPUnit_Extensions_Database_TestCase takes forever to run. With 189 tests, the suite takes around 8-9 minutes. I was kind of hoping it would take 30 seconds at most ;-)

It looks like restoring the database to its initial state is the process that takes the time, so we've made our datasets as small as possible and limit the number of tables we require for each test case. I am using fixtures and sharing as much as possible.

Are there any settings or modifications I can use to speed the execution up? Looking at what the MySQL server is doing throughout the tests it seems that lots of truncate/inserting is happening, but surely it would be faster to pack the test data sets into temporary tables and then simply select into them for each test?

The driver I'm using is PDO/MySQL with an XML test dataset.

Flaming answered 30/4, 2012 at 15:14 Comment(4)
You need to metric where the bottleneck is. Probably it will speed up your needs if you can mock the whole database so you don't need to run dbunit at all. A test should run under a 10th of second - which is actually considerable slow for a test.Tailor
Since you have not mentioned, are you using Fixtures and sharing as much as possible?Stagey
I'm using fixtures and sharing as much as reasonably possible. Is there a way I can profile the test runner?Flaming
You can exclude the test files you are not using currently.Escalade
F
22

Upon Googling I've managed to reduce the time it takes from 10 minutes down to 1 minute. It turns out that changing some InnoDB configuration settings in my.ini/my.cnf will help.

Setting innodb_flush_log_at_trx_commit = 2 seems to do the job. After you change it, restart your MySQL server.

More on dev.mysql.com: innodb_flush_log_at_trx_commit

The setting controls how ACID compliant the flushing of the logs is. The default value is 1 which is full ACID compliance which means

the log buffer is written out to the log file at each transaction commit and the flush to disk operation is performed on the log file.

With a value of 2, the following happens:

The log buffer is written out to the file at each commit, but the flush to disk operation is not performed on it.

The key difference here is that because the log isn't written out at every commit, an operating system crash or power outage can wipe it out. For production, stick to a value of 1. For local development with a test database, the value of 2 should be safe.

If you're working with data that will be transferred to the live database, I would suggest sticking with the value of 1.

Flaming answered 2/5, 2012 at 9:48 Comment(4)
Many thanks - this cut my time taken by 90% too! Strange though since my tables are MyISAM... Maybe it is the internal tables?Cremona
Yeah, this seems to do wonders... just be sure to put it under the [mysqld] section (not anywhere else).Sandwich
This does not really help by pgsql :-( Btw I think dbunit sux, it's extremely slow..Escalade
read the docs, it's really important to understand the difference in the values. use the value of 2 for local development never for production. Before changing the value it took ~51 seconds to run 26 tests. Now it takes ~2 seconds. Too bad you can't change the setting for specific databases.Verenaverene
E
1

The fixture creation in DbUnit is extremely slow. It takes 1.5 sec every time with core2duo e8400 4gb kingston 1333. You can find the bottleneck with xdebug and fix it (if you can), or you can do one of the following:

1.)

You can run only test files you currently develop with a custom bootstrap xml:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://phpunit.de/phpunit.xsd"
         backupGlobals="false"
         verbose="true"
         bootstrap="test/bootstrap.php">
    <testsuites>
        <testsuite>
            <directory>test/integration</directory>
            <exclude>test/integration/database/RoleDataTest.php</exclude>
        </testsuite>
    </testsuites>
    <php>
        <env name="APPLICATION_MODE" value="test"/>
    </php>
</phpunit>

The exclude part is important here. You can use test groups too.

2.)

namespace test\integration;


abstract class AbstractTestCase extends \PHPUnit_Extensions_Database_TestCase
{
    static protected $pdo;
    static protected $connection;

    /**
     * @return \PHPUnit_Extensions_Database_DB_IDatabaseConnection
     */
    public function getConnection()
    {
        if (!isset(static::$pdo)) {
            static::$pdo = new \PDO('pgsql:host=localhost;port=5432;dbname=dobra_test', 'postgres', 'inflames', array(\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION));
            static::$connection = $this->createDefaultDBConnection(static::$pdo);
        }
        return static::$connection;
    }

    /**
     * @return \PHPUnit_Extensions_Database_Operation_DatabaseOperation
     */

    static protected $fixtureSet = false;

    protected function getSetUpOperation()
    {
        $c = get_class($this;
        if (!$c::$fixtureSet) {
            $c::$fixtureSet = true;
            return \PHPUnit_Extensions_Database_Operation_Factory::CLEAN_INSERT(true);
        }
        return \PHPUnit_Extensions_Database_Operation_Factory::NONE();
    }

    static protected $dataSet;

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    public function getDataSet()
    {
        $c = get_class($this;
        if (!isset($c::$dataSet)) {
            $c::$dataSet = $this->createDataSet();
        }
        return $c::$dataSet;
    }

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    abstract protected function createDataSet();

    protected function dataSetToRows($tableName, array $ids)
    {
        $transformer = new DataSetRowsTransformer($this->getDataSet());
        $transformer->findRowsByIds($tableName, $ids);
        $transformer->cutColumnPrefix();
        return $transformer->getRows();
    }

}

You can override the TestCase. In this example you will use only one pdo connection by every test case (you can inject it to your code with dependency injection), by overriding setup operation you can set the fixture only once per testcase or only once for every test (depends on self:: or $cls = get_class($this); $cls::). (PHPUnit has bad design, it creates new instance by every test call, so you have to hack with the class names to store variables per instance or per class.) By this scenario you have to write the tests to depend on eachother with @depend annotation. For example you can delete the same row you created in the previous test.

By this test code 1.5 secs instead of 6 x 1.5 = 9 secs:

namespace test\integration\database;

use Authorization\PermissionData;
use test\integration\AbstractTestCase;
use test\integration\ArrayDataSet;

class PermissionDataTest extends AbstractTestCase
{
    static protected $fixtureSet = false;
    static protected $dataSet;

    /** @var PermissionData */
    protected $permissionData;

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    public function createDataSet()
    {
        return new ArrayDataSet(array(
            'permission' => array(
                array('permission_id' => '1', 'permission_method' => 'GET', 'permission_resource' => '^/$'),
                array('permission_id' => '2', 'permission_method' => 'POST', 'permission_resource' => '^/$'),
                array('permission_id' => '3', 'permission_method' => 'DELETE', 'permission_resource' => '^/$')
            ),
            'user' => array(
                array('user_id' => '1', 'user_name' => 'Jánszky László', 'user_email' => '[email protected]', 'user_salt' => '12435')
            ),
            'user_permission' => array(
                array('user_permission_id' => '1', 'user_id' => '1', 'permission_id' => '1'),
                array('user_permission_id' => '2', 'user_id' => '1', 'permission_id' => '2')
            ),
            'role' => array(
                array('role_id' => '1', 'role_name' => 'admin')
            ),
            'role_permission' => array(
                array('role_permission_id' => '1', 'role_id' => '1', 'permission_id' => '1')
            ),
            'permission_cache' => array(
                array('permission_cache_id' => '1', 'user_id' => '1', 'permission_id' => '1'),
                array('permission_cache_id' => '2', 'user_id' => '1', 'permission_id' => '2'),
            )
        ));
    }

    public function testReadAllShouldReturnEveryRow()
    {
        $this->assertEquals($this->permissionData->readAll(), $this->dataSetToRows('permission', array(3, 2, 1)));
    }

    /** @depends testReadAllShouldReturnEveryRow */

    public function testReadAllByRoleIdShouldReturnEveryRowRelatedToRoleId()
    {
        $this->assertEquals($this->permissionData->readAllByRoleId(1), $this->dataSetToRows('permission', array(1)));
    }

    /** @depends testReadAllByRoleIdShouldReturnEveryRowRelatedToRoleId */

    public function testReadAllByUserIdShouldReturnEveryRowRelatedToUserId()
    {
        $this->assertEquals($this->permissionData->readAllByUserId(1), $this->dataSetToRows('permission', array(2, 1)));
    }

    /** @depends testReadAllByUserIdShouldReturnEveryRowRelatedToUserId */

    public function testCreateShouldAddNewRow()
    {
        $method = 'PUT';
        $resource = '^/$';
        $createdRow = $this->permissionData->create($method, $resource);
        $this->assertTrue($createdRow['id'] > 0);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount() + 1, $this->getConnection()->getRowCount('permission'));
        return $createdRow;
    }

    /** @depends testCreateShouldAddNewRow */

    public function testDeleteShouldRemoveRow(array $createdRow)
    {
        $this->permissionData->delete($createdRow['id']);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount(), $this->getConnection()->getRowCount('permission'));
    }

    /** @depends testDeleteShouldRemoveRow */

    public function testDeleteShouldRemoveRowAndRelations()
    {
        $this->permissionData->delete(1);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount() - 1, $this->getConnection()->getRowCount('permission'));
        $this->assertEquals($this->getDataSet()->getTable('user_permission')->getRowCount() - 1, $this->getConnection()->getRowCount('user_permission'));
        $this->assertEquals($this->getDataSet()->getTable('role_permission')->getRowCount() - 1, $this->getConnection()->getRowCount('role_permission'));
        $this->assertEquals($this->getDataSet()->getTable('permission_cache')->getRowCount() - 1, $this->getConnection()->getRowCount('permission_cache'));
    }

    public function setUp()
    {
        parent::setUp();
        $this->permissionData = new PermissionData($this->getConnection()->getConnection());
    }
}

3.)

Another solution to create the fixture only once per project, and after that use every test in transactions and rollback after every test. (This does not work if you have pgsql deferred code which needs commit to check the constraints.)

Escalade answered 11/7, 2013 at 20:13 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.