Optimising behat test suite
Asked Answered
H

2

3

I have a test suite which has 20 feaure files and 100% MySQL CRUD operations are being carried out. It takes about 5 minutes to finish. If I did the test manually it would take about 7 minutes max. What I need to know is, what I need to do in order to optimise the whole process?

Note: ParallelRunnder is not supported for Behat 3 so it is out of scope for now!

If you're going to suggest to use Behat 3, then please help me modifying my composer.json & behat.yml files because when I do it myself and run bin/behat I get errors like:

`Behat\Symfony2Extension\Extension` extension file or class could not be located.
`Behat\MinkExtension\Extension` extension file or class could not be located. 
Unrecognized options "mink_driver" under "testwork.symfony2"
Unrecognized options "context, paths" under "testwork" 

As you can see below, I use certain-ish versions numbers, no star signs.

CURRENT composer.json:

"require": {
    "php": ">=5.3.3",
    "symfony/symfony": "2.5.4",
    "doctrine/orm": "~2.2,>=2.2.3",
    "doctrine/doctrine-bundle": "~1.2",
    "twig/extensions": "~1.0",
    "symfony/assetic-bundle": "~2.3",
    "symfony/swiftmailer-bundle": "~2.3",
    "symfony/monolog-bundle": "~2.4",
    "sensio/distribution-bundle": "~3.0",
    "sensio/framework-extra-bundle": "~3.0",
    "incenteev/composer-parameter-handler": "~2.0",
    "behat/behat": "2.5.3",
    "behat/behat-bundle": "1.0.0",
    "behat/symfony2-extension": "1.1.2",
    "behat/mink": "1.5.0",
    "behat/mink-extension": "~1.3",
    "behat/mink-selenium2-driver": "1.1.1",
    "behat/mink-goutte-driver": "1.0.9"
},

CURRENT behat.yml:

default:
    context:
        class: FeatureContext
        parameters:
            output_path: %behat.paths.base%/build/behat/output/
            screen_shot_path: %behat.paths.base%/build/behat/screenshot/
    extensions:
        Behat\Symfony2Extension\Extension:
            mink_driver: true
            kernel:
                env: test
                debug: true
        Behat\MinkExtension\Extension:
            base_url: 'http://symfony.local/app_test.php/'
            files_path: %behat.paths.base%/build/dummy/
            javascript_session: selenium2
            browser_name: firefox
            goutte: ~
            selenium2: ~
    paths:
        features: %behat.paths.base%/src
        bootstrap: %behat.paths.features%/Context

EDIT:

I have 20 feature files and one scenario in each. For the CRUD operations:

  • I have login method running in each scenario. Query runs against DB.
  • Some scenarios do INSERT, some to UPDATE and some do DELETE.
  • I'm using pdo_mysql as database_driver

A part of my FeatureContext file:

namespace Site\CommonBundle\Features\Context;

use Behat\MinkExtension\Context\MinkContext;
use Behat\Mink\Exception\UnsupportedDriverActionException;
use Behat\Mink\Driver\Selenium2Driver;
use Behat\Symfony2Extension\Context\KernelAwareInterface;
use Symfony\Component\HttpKernel\KernelInterface;
use Symfony\Component\Process\Process;

/**
 * Class FeatureContext
 *
 * Parent to other FeatureContext files. It is used to avoid duplicated codes and all the
 * shared commons are kept here.
 *
 * @package Site\CommonBundle\Features
 */
class FeatureContext extends MinkContext implements KernelAwareInterface
{
    protected $kernel;
    protected $screenShotPath;
    protected $outputPath;

    /**
     * Parameter $parameters comes from behat.yml file.
     * @param array $parameters
     */
    public function __construct(array $parameters)
    {
        $this->outputPath = $parameters['output_path'];
        $this->screenShotPath = $parameters['screen_shot_path'];
    }

    /**
     * Helps to use doctrine and entity manager.
     * @param KernelInterface $kernelInterface Interface for getting Kernel.
     */
    public function setKernel(KernelInterface $kernelInterface)
    {
        $this->kernel = $kernelInterface;
    }

    /**
     * Without this, PhantomJs will fail if responsive design is in use.
     * @BeforeStep
     */
    public function resizeWindow()
    {
        $this->getSession()->resizeWindow(1440, 900, 'current');
    }

    /**
     * Take screen-shot when step fails. Works only with Selenium2Driver.
     *
     * @AfterStep
     * @param $event Current event.
     * @throws \Behat\Mink\Exception\UnsupportedDriverActionException
     */
    public function takeScreenshotAfterFailedStep($event)
    {
        if (4 === $event->getResult()) {
            $driver = $this->getSession()->getDriver();

            if (! ($driver instanceof Selenium2Driver)) {
                throw new UnsupportedDriverActionException(
                    'Taking screen-shots is not supported by %s, use Selenium2Driver instead.',
                    $driver
                );

                return;
            }

            if (! is_dir($this->screenShotPath)) {
                mkdir($this->screenShotPath, 0777, true);
            }

            $filename = sprintf(
                '%s_%s_%s.%s',
                $this->getMinkParameter('browser_name'),
                date('Ymd') . '-' . date('His'),
                uniqid('', true),
                'png'
            );

            file_put_contents($this->screenShotPath . '/' . $filename, $driver->getScreenshot());
        }
    }

    /**
     * @When /^I login as "([^"]*)"$/
     * @param $type User role type.
     */
    public function iLoginAs($type)
    {
        $container = $this->kernel->getContainer();
        $userData = $container->getParameter('dummy_user');

        $this->visit('/login');
        $this->fillField('username', $userData[$type]['username']);
        $this->fillField('password', $userData[$type]['password']);
        $this->pressButton('_submit');
    }
    .........
}
Homeward answered 16/10, 2014 at 16:15 Comment(6)
Tell a bit more about the features and scenarios, how many scenarios in total? What hooks and db setup operations do you have? When / for what tests are you using Selenium?Rinna
I updated the post but didn't quiet get this question - When / for what tests are you using Selenium?Homeward
You have Goutte and Selenium driver, you have Firefox set as your browser and there are some comments about PhantomJS. I assume you are using Selenium driver for some specific tests or just for everything?Rinna
I have a build.xml so If I call bin/phing behat-selenium, it will run all with selenium, if I call bin/phing behat-phantomjs, it will run all with pjantomjs. The reason why PhantomJS is because it is much faster compared to Selenium. Halfs the time!Homeward
The Shard is shining and it is 6pm now. Time to go!Homeward
Oh dear… I miss the Shard. Wave at it on the way back!Rinna
R
9

I don't know why I thought it takes 20 minutes to run, probably confused that with the number of features. 5 minutes is not bad at all. There are some basic things you can do that can help to speed it up.

  1. Logic inside your @BeforeStep – you probably can move it into @BeforeScenario or even into @BeforeFeature or even into @BeforeSuite – there's no need to do that so often.
  2. Very obvious, but just in case: Goutte driver is amazingly fast compared to others, the negative is that it doesn't support JS and you can't take screenshots. But I take it the Symfony app doesn't involve much of JS to do CRUD operations. So, double check everything that doesn't involve JS and use it with Goutte.
  3. iLoginAs is probably used in every step. You can update this to be much faster by manually creating a user session and setting back the cookie as a header, i.e., the same logic from your authentication method goes in the step definition and then you simply do $this->getSession()->getDriver()->setCookie(session_name(), session_id()); – on the next request your user is already authenticated.
  4. browser_name: firefox – god no… use chrome, it must be faster.
  5. Using HHVM for running Behat tests (the app still uses the old good PHP) will give a good increase in performance. Might be a pain installing it, especially on Mac.
  6. Finally, do migrate to Behat 3, that shouldn't be painful at all. They've taken a lot of knowledge from 2.x and used it to improve on it, including performance aspects. It's also much more flexible on the configuration, allows multiple contexts to keep things better organised and cleaner.
Rinna answered 16/10, 2014 at 17:45 Comment(1)
2,3,5,3 need time for reseach so I'll do all that over the weekend. Thank u for the well exlained info. +1Homeward
H
5

I'll publish outcomes here after implementing all the suggestions above for others to see how they worked out so everytime I cover one step I'll update this post.

ORIGINAL TIME:

Total time: 4 minutes  12.55 seconds

TIME AFTER IMPLEMENTING STEP 1:

Total time: 2 minutes  8.01 seconds

Note: Only @BeforeScenario seems to be working because accessing $this key in static methods is not allowed in OOP since @BeforeFeature and @BeforeSuite requires resizeWindow() to be static. If it was possible, result would be much much faster.

TIME AFTER IMPLEMENTING STEP 2:

Note: Read step 6 below.

TIME AFTER IMPLEMENTING STEP 3:

Note: I'm a Symfony2 user and unfortunatelly failed to implement this point.

TIME AFTER IMPLEMENTING STEP 4:

Total time: 1 minutes  54.11 seconds

Note: Behat 3 users - run selenium with java -jar selenium-server-standalone-2.43.1.jar -Dwebdriver.chrome.driver="chromedriver"

TIME AFTER IMPLEMENTING STEP 6:

Total time: 0 minutes  52.04 seconds

Note: Symfony2 users - if you set default_session to symfony2, it is bleeding fast compared to goutte and selenium2. Goutte: 1 minutes 20.03 seconds, Selenium2: 1 minutes 31.00 seconds

Homeward answered 17/10, 2014 at 12:55 Comment(2)
Try this for step 3: robinvdvleuten.nl/blog/handle-authenticated-users-in-behat-minkHuntley
@Huntley - If I have 10 I am authenticated as .... scenarios in my feature, the method iAmAuthenticatedAs() would get triggered 10 times so as a result it would login 10 times, wouldn't it? Or only once and read from session/cookie?Homeward

© 2022 - 2024 — McMap. All rights reserved.