Running behat tests in parallel (in two browser windows)

Question

I followed this blog as an example and read the ParallerRunner info. When I call bin/behat command, one browser window opens and runs all the tests successfully with the setting below.

symfony/behat.yml

default:
    context:
        class: Site\CommonBundle\Features\Context\FeatureContext
    extensions:
        Behat\Symfony2Extension\Extension:
            mink_driver: true
            kernel:
                env: test
                debug: true
        Behat\MinkExtension\Extension:
            base_url: 'http://symfony.local/app_test.php/'
            javascript_session: selenium2
            browser_name: firefox
            goutte: ~
            selenium2: ~
    paths:
        features: %behat.paths.base%/src
        bootstrap: %behat.paths.features%/Context

I modified the behay.yml (as shown below) to run some tests in one browser window and some in another window, however it doesn't do that. What it does is, it opens two browser windows but they both run same tests! How can I overcome this issue?

symfony/behat.yml

default:
    context:
        class: Site\CommonBundle\Features\Context\FeatureContext
        parameters:
            output_path: %behat.paths.base%/build/behat/output/
            screen_shot_path: %behat.paths.base%/build/behat/screenshot/
    extensions:
        Behat\Symfony2Extension\Extension:
            mink_driver: true
            kernel:
                env: test
                debug: true
        Behat\MinkExtension\Extension:
            base_url: 'http://symfony.local/app_test.php/'
            files_path: %behat.paths.base%/build/dummy/
            javascript_session: selenium2
            browser_name: firefox
            goutte: ~
            selenium2: ~
        shvetsgroup\ParallelRunner\Extension:
            process_count:  2
    paths:
        features: %behat.paths.base%/src
        bootstrap: %behat.paths.features%/Context

F1:
    filters:
        tags: "@backend"
F2:
    filters:
        tags: "@frontend"

BEHAT TESTS:

This should run in one window:

@frontend
Feature: User Login

  @javascript
  Scenario: I can login to the system
    Given I am on "/login"
    And I login as "user"

This should run in another window:

@backend
Feature: Admin Login

  @javascript
  Scenario: I can login to the system
    Given I am on "/login"
    And I login as "admin"

Jussive

I did see this yesterday but I never dealt with the parallel tests before. I've read about the theory and to me it seems problematic, as you need a database and server setup for each process – I don't know if that applies to your case, but it is true for most situations. That's an interesting question, I'd love to see an answer for that. – Sourdough

@IanBytchek - No worries. I'll sit and wait. If this is not possible yet then someone should make it possible in near future because there is no point of waiting for long time for a test suite to finish when one/a few test(s) do not depend on another. Sounds like the ParallelRunner is not being actively developed anyway. – Jussive

You could try upgrading to Behat 3 in the meanwhile, maybe that will bring some luck? :) Parallelisation is a great thing, Behat kills my nerves with the time it takes to run the tests. Totally share the pain. – Sourdough

@IanBytchek - ParallelRunner didn't like Behat 3 :) shvetsgroup/parallelrunner dev-master requires behat/behat >=2.4.6,<3.0 -> satisfiable by behat/behat[2.5.x-dev].. Of course, one is being developed actively and another isn't! – Jussive

Goddammit… github.com/shvetsgroup/ParallelRunner/issues/26 – Sourdough

While I was writing my note there, your note appeared suddenly :) – Jussive

I've read at the saucelabs about the whole concept. I assume you are not looking for a paid subscription, nor I actually tried this myself, nor I'm actually recommending it :) but the minimum plan is 12/mo. Maybe it's something to consider? Besides, a lot of stuff can be optimised often. If your suite takes half an hour I'd say something is fishy about it. Create another question with some details of what's going on, I'll throw some thoughts at it. I have a pretty large and intense suite, takes only two mins. – Sourdough

There is is.... – Jussive

Answer 1

I setup parallel test execution with GNU Parallel and xargs. Also implement consolidated report for all executed features. Details in my article here:
http://parallelandvisualtestingwithbehat.blogspot.com/p/blog-page.html

Bohon

Answer 2

find ./features -name "*.feature" |
  parallel --gnu --halt-on-error=0 -j 3 --keep-order vendor/bin/behat -c src/my_directory/behat.yml

--halt-on-error possibilities are :

• 0 Do not halt if a job fails. Exit status will be the number of jobs failed. This is the default.

• 1 Do not start new jobs if a job fails, but complete the running jobs including cleanup. The exit status will be the exit status from the last failing job.

• 2 Kill off all jobs immediately and exit without cleanup. The exit status will be the exit status from the failing job.

-j 3 : Run 3 jobs in parallel

It work perfectly with selenium.

Presuppose