Behat hangs when there are multiple scenarios, but works on a single one

Question

I have Behat test cases written like so:

Feature: Checkout
In order to buy products
As a customer
I need to be able to checkout items in the cart

Background: 
    Given step 1
    And step 2

@Ready
Scenario: Deliver now
    When step 3
    Then step 4

@NoneReady
Scenario: Deliver later
    When step a
    Then step b
    And step c


@AddressNotCovered
Scenario: Address Not Covered
    When step i
    Then step ii

If I run behat on a single tag, it works just fine:

$ behat --tags=Ready
Feature: Checkout
  In order to buy products
  As a customer
  I need to be able to checkout items in the cart

  @Ready
  Scenario: Deliver now                                                                 # tests/features/Checkout/CheckOut.feature:9
    step 1
    And step 2 
    .. 

1 scenario (1 passed)
7 steps (7 passed)
0m3.85s (36.62Mb)

But if I run it on multiple tags, it hangs at the end of the first tag:

behat --tags=Ready,AddressNotCovered
Feature: Checkout
  In order to buy products
  As a customer
  I need to be able to checkout items in the cart

  @Ready
  Scenario: Deliver now                                                                 # tests/features/Checkout/CheckOut.feature:9
    Given step ..
    ..
    And .. 

    // hangs here

What am I doing wrong?

Environment

Laravel 5.4
Behat 3.1.0
PHP 7.1.23
PHPUnit 5.7.27

from my composer.json

"require": {
    "php": ">=5.5.9",
    "laravel/framework": "5.4.*",
    ..
    "behat/behat": "3.1.0",
    "laracasts/behat-laravel-extension": "^1.1",
},
"require-dev": {
    "phpunit/phpunit": "~5.7",
    "phpspec/phpspec": "~2.1",
    "johnkary/phpunit-speedtrap": "^1.0",
},

Behat.yml

default:
  extensions:
      Laracasts\Behat:
          env_path: .env.testing
  autoload:
    - ./tests/features/bootstrap
  suites:

    Checkout:
      paths: [./tests/features/Checkout]
      contexts: [CheckoutFeatureContext]

---

Update

I tried to create sample gherkin to illustrate the problem above. I ran into the same problem when trying to auto append snippets. Appending snippets worked with a single scenario, but failed on multiple scenarios:

working example: single scenario

# tests/features/Example/Example.feature

Feature: Example
In order to show dev team how to use behat/gherkin using background 
As a developer
I need to be able write gherkin using a background and multiple scenarios 
And all scenarios should run 

Background: 
    Givens setup condition 1 
    And setup condition 2 


Scenario: scenario one 
    When I perform first sample trigger point 
    Then result one must happen 
    And result two must happen 

When I run the following command

behat tests/features/Example/Example.feature  --append-snippets

adding snippets worked just fine

Feature: Example
  In order to show dev team how to use behat/gherkin using background
  As a developer
  I need to be able write gherkin using a background and multiple scenarios
  And all scenarios should run

  Background:             # tests/features/Example/Example.feature:9
      Givens setup condition 1
    And setup condition 2

  Scenario: scenario one                      # tests/features/Example/Example.feature:13
    When I perform first sample trigger point
    Then result one must happen
    And result two must happen

1 scenario (1 undefined)
4 steps (4 undefined)
0m0.48s (24.63Mb)

u tests/features/bootstrap/FeatureContext.php - `setup condition 2` definition added
u tests/features/bootstrap/FeatureContext.php - `I perform first sample trigger point` definition added
u tests/features/bootstrap/FeatureContext.php - `result one must happen` definition added
u tests/features/bootstrap/FeatureContext.php - `result two must happen` definition added

failing example: multiple scenarios

when we have multiple scenarios

# tests/features/Example/Example.feature

Feature: Example
In order to show dev team how to use behat/gherkin using background 
As a developer
I need to be able write gherkin using a background and multiple scenarios 
And all scenarios should run 

Background: 
    Givens setup condition 1 
    And setup condition 2 

Scenario: scenario one 
    When I perform first sample trigger point 
    Then result one must happen 
    And result two must happen 

Scenario: scenario two 
    When I perform second sample trigger point 
    Then result a must happen 
    And result b must happen 

running the same --append-snippets command chokes:

Feature: Example
  In order to show dev team how to use behat/gherkin using background
  As a developer
  I need to be able write gherkin using a background and multiple scenarios
  And all scenarios should run

  Background:             # tests/features/Example/Example.feature:9
      Givens setup condition 1
    And setup condition 2

  Scenario: scenario one                      # tests/features/Example/Example.feature:13
    When I perform first sample trigger point
    Then result one must happen
    And result two must happen

^C // had to abort here

Gristmill

Share the FeatureContext / Suite definition Class, too. Most likely - there is "the problem". – Bridie

@Bridie I shared the behat.yml, but that's not what you're asking for. I can't share my unit test code b/c it exposes a lot of business logic. That said you gave me an idea: I'll create basic dummy tests and try to use it to repro the problem. If the problem persists I'll just share that code – Gristmill

Very good idea! – Bridie

@Bridie updated the question with sample code.. i couldn't even generate the snippets! – Gristmill

@Bridie if you can show me sample gherkin that runs with multiple scenarios (either actually running the code or adding snippets) that would be enough to award you correct answer – Gristmill

Answer 1

It turns out the examples above were too simplistic. After doing some research (especially helpful was this post) I realized that this "stalling" is due to tearing down the database after each test. So this is what fixed it:

First I replaced DatabaseTransactions with DatabaseMigrations in my FeatureContext class:

class FeatureContext extends TestCase implements  Context, SnippetAcceptingContext
{

    use DatabaseMigrations, ..

Given the above, I removed the manual migration comand from my bitbucket pipeline script

- php artisan --env=testing config:cache

which makes sense since with the new code, the database will always be refreshed and migrated before each test.

Then I added the setUp() call to the behat hooks:

/** @BeforeScenario */
public function before(BeforeScenarioScope $scope)
{
    parent::setUp();
}

And that's it. The best part about this solution is that it completely aligned my local testing environment with that of bitbucket pipelines, so that the results were always the same.

Further explanation: from our wiki

In general, it's a good idea to start each test fresh without left overs from the previous test (esp when it comes to databases). In the words of laravel:

It is often useful to reset your database after each test so that data from a previous test does not interfere with subsequent tests.

For that we use migrations. That being said, since we're actually using Behat, we need this migration to happen before and after each scenario life cycle. We do that using Behat's hooks. We do that here:

/** @BeforeScenario */
    public function before(BeforeScenarioScope $scope)
    {
        parent::setUp();
    }

parent::setUP() tells the Laravel framework to do the necessary work before and after each scenario:

 protected function setUp()
    {
        if (! $this->app) {
            $this->refreshApplication();
        }
        $this->setUpTraits(); <---- here
        ..

This in turn calls setup traits:

   protected function setUpTraits()
    {
        $uses = array_flip(class_uses_recursive(static::class));
        if (isset($uses[DatabaseMigrations::class])) {
            $this->runDatabaseMigrations();
        }
        ..

which calls this

public function runDatabaseMigrations()
{
    $this->artisan('migrate:fresh');
    $this->app[Kernel::class]->setArtisan(null);
    $this->beforeApplicationDestroyed(function () {
        $this->artisan('migrate:rollback');
        RefreshDatabaseState::$migrated = false;
    });
}

Notice that Laravel will also rollback the changes once the application is destroyed. It's very important to understand this in order to prevent Behat stalling when there are multiple scenarios and a given before them. Also keep in mind that when we use Gherkin like so:

Feature: Checkout
In order to buy products
As a customer
I need to be able to checkout items in the cart

Background: 
    Given step 1
    And step 2

@Ready
Scenario: Deliver now
    When step 3
    Then step 4

@NoneReady
Scenario: Deliver later
    When step a
    Then step b
    And step c

Then each scenario starts with the background steps, not in the scenario steps itself

Example:

Feature: Checkout
In order to buy products
As a customer
I need to be able to checkout items in the cart

Background:
    Given step 1  <-- every scenario starts here, so we call setup before this step
    And step 2

@Ready
Scenario: Deliver now
    When step 3 <-- not here
    Then step 4

@NoneReady
Scenario: Deliver later
    When step a
    Then step b
    And step c

Gristmill

Answer 2

The following steps work for me on Arch Linux and PHP 7.3:

composer global require laravel/installer
laravel new behat-laravel
cd behat-laravel
composer require behat/behat behat/mink behat/mink-extension laracasts/behat-laravel-extension --dev
touch behat.yml
# edit behat.yml
# edit features/bootstrap/FeatureContext.php
vendor/bin/behat --init
# Add steps and tag them
vendor/bin/behat
vendor/bin/behat --tags Ready,NoneReady

# behat.yml
default:
    extensions:
        Laracasts\Behat:
            # env_path: .env.behat
        Behat\MinkExtension:
            default_session: laravel
            laravel: ~

<?php
// features/bootstrap/FeatureContext.php

use Behat\Behat\Hook\Scope\AfterStepScope;
use Behat\Behat\Tester\Exception\PendingException;
use Behat\Behat\Context\Context;
use Behat\Behat\Context\SnippetAcceptingContext;
use Behat\Gherkin\Node\PyStringNode;
use Behat\Gherkin\Node\TableNode;
#This will be needed if you require "behat/mink-selenium2-driver"
#use Behat\Mink\Driver\Selenium2Driver;
use Behat\MinkExtension\Context\MinkContext;

/**
 * Defines application features from the specific context.
 */
class FeatureContext extends MinkContext implements Context, SnippetAcceptingContext
{
    /**
     * Initializes context.
     *
     * Every scenario gets its own context instance.
     * You can also pass arbitrary arguments to the
     * context constructor through behat.yml.
     */
    public function __construct()
    {
    }

    /**
     * @When stepready :arg1
     */
    public function stepready($arg1)
    {
        return true;
    }

    /**
     * @When steplater :arg1
     */
    public function steplater($arg1)
    {
        return true;
    }

}

# feature/customer.feature
Feature: Checkout
  In order to buy products
  As a customer
  I need to be able to checkout items in the cart

  @Ready
  Scenario: Deliver now 
    When stepready 1
    Then stepready 2 

  @NoneReady
  Scenario: Deliver later
    When steplater 1
    Then steplater 2 

frosch ➜ behat-laravel vendor/bin/behat --tags Ready,NoneReady
Feature: Checkout
  In order to buy products
  As a customer
  I need to be able to checkout items in the cart

  @Ready
  Scenario: Deliver now # features/customer.feature:7
    When stepready 1    # FeatureContext::stepready()
    Then stepready 2    # FeatureContext::stepready()

  @NoneReady
  Scenario: Deliver later # features/customer.feature:12
    When steplater 1      # FeatureContext::steplater()
    Then steplater 2      # FeatureContext::steplater()

2 Szenarien (2 bestanden)
4 Schritte (4 bestanden)
0m0.02s (18.47Mb)

Bridie

why mink? is it necessary? – Gristmill

we don't do any browser testing – Gristmill

I just followed the docs on the laravel plugin site. So you not use any http client? – Bridie

i do we just don't unit test that yet.. anyways i followed your steps and they worked just fine (see the github repo i created) I just have to see now why it's not working on my project – Gristmill

it may have to do with the fact that your code uses laravel 5.8 and behat 3.5 rather than what i have in my project – Gristmill

I am have been using Behat since 2014 and Version 2. I have never encountered above problem "hanging" etc. Hard to remotely debug. Maybe try bumping Behat to the latest version? Maybe it already is in composer.lock... – Bridie

Also: make sure you are using the Behat script from the vendor directory. – Bridie

Let us continue this discussion in chat. – Gristmill