continue running cucumber steps after a failure

Question

Is there any way to continue executing Cucumber Steps even when one of the steps fails. In my current setup when a step fails , cucumber skips remaining steps....I wonder if there is some way to twick cucumber runner setup..

I could comment out failing steps but its not practical when you dont know which step will fail...If i could continue with remaining step i would know complete set of failing Tests in one shot....rather than going in cycle over cycle...

Environment: Cucumber JVM , R , Java , Ibatis , Spring Framework, Maven

Clone

Answer 1

It is not a good idea to continue executing steps after a step failure because a step failure can leave the World with an invariant violation. A better strategy is to increase the granularity of your scenarios. Instead of writing a single scenario with several "Then" statements, use a list of examples to separately test each postconditions. Sometimes a scenario outline and list of examples can consolidate similar stories. https://docs.cucumber.io/gherkin/reference/#scenario-outline

There is some discussion about adding a feature to tag certain steps to continue after failure. https://github.com/cucumber/cucumber/issues/79

Faithless

I see this is from years ago but if you can, could you please update the broken link? – Ware

"because a step failure can leave the World with an invariant violation". That assumption is true but not always the case. There are many instances where I need to a check a value and just because the value is incorrect that doesn't mean the rest of the scenario can not be run. – Outsole

Answer 2

One way would be to catch all the assertion errors and decide in the last step whether to fail or pass the test case. In this case, you can tailor it, say, check at any step to see if there is more than n errors and fail the test, if so.

Here's what I have done:

1. initialize a StringBuffer for Errors in your @Before for the test cases
2. catch the Assertion Errors and add to the StringBuffer, so that, they do not get thrown and terminate the test case.

3. Check the StringBuffer to determine whether to fail the test case.

   StringBuffer verificationErrors;
   
   // Initialize your error SringBuffer here
   
   @Before
   public void initialize() {
       verificationErrors = new StringBuffer();
   }
   // The following is one of the steps in the test case where I need to assert     something
   
   @When("^the value is (\\d+)$")
   public void the_result_should_be(int arg1) {
       try  {
           assertEquals(arg1, 0);
       }
       catch(AssertionError ae) {
           verificationErrors.append("Value is incorrect- "+ae.getMessage());  
       }
   

Check the StringBuffer in @After or in the last step of test case to determine if you can pass it or fail it, as follows:

if (!(verificationErrors.size()==0)) {
   fail(verificationErrors.toString());
}

The only issue would be that, in the report, all the steps would look green but the test case looks failed. Then you might have to look through the Errors String to know which step(s) failed. You could add extra information to the String whenever there is an Assertion Error to help you locate the step easily.

Reluctance

"The only issue would be that, in the report, all the steps would look green but the test case looks failed." That's exactly why you don't want to do this. – Cyton

Answer 3

Use SoftAssert to accumulate all assertion failures. Then tag your step definitions class as @ScenarioScoped and in step definitions class add a method tagged @After where you do mySoftAssert.assertAll();

i.e.

import io.cucumber.guice.ScenarioScoped;
import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.en.Then;

@ScenarioScoped
public class MyStepDefinitions {

SoftAssert mySoftAssert=new SoftAssert();

    @Then("check something")
    public void checkSomething() {
       mySoftAssert.assertTrue(actualValue>expectedMinValue);
    }

    @After
    public void afterScenario(Scenario scenario) throws Exception {
        mySoftAssert.assertAll();
    }
}

Retentivity