How to rerun the failed scenarios using Cucumber?

Question

I'm using Cucumber for my tests. How do I rerun only the failed tests?

Caftan

Answer 1

Run Cucumber with rerun formatter:

cucumber -f rerun --out rerun.txt

It will output locations of all failed scenarios to this file.

Then you can rerun them by using

cucumber @rerun.txt

Sholom

it there anyway of putting it all in one cmd line? – Susurrate

You can use a rake task for run both cucumber commands. Afterwards you can call the rake file as: rake features:jenkins_with_rerun – Owades

You can use && to put it all in one cmd line (if you want to just start it and walk away) cucumber -f rerun --out rerun.txt && cucumber @rerun.txt – Hailstorm

@JoeSusnick you can't use && because it won't rerun if the first command fails (which is when you need it!) You want || (to only rerun on failure) or ; (to always rerun). – Oink

Also, the file is not read first, but written first. So if you would like to chain cucumber command calls, with each generating a rerun.txt file using that format, that won't simply work =/. You will have to alternate the filename or use some other strategy for rerunning tests quickly. – Fulgurating

In 2016, how would this work with Protractor since I run my tests with "protractor protractor.conf.js" ? Lastly, I'm using cucumberJS, the Javascript implementation of cucumber. – Lovelovebird

rerun.txt text holds the feature file name of the failed scenario. If i have two scenarios in a feature file and one of it fails, does the rerun do on feature level making the passed one to run again, and it is not on the scenario level, as i understood. Is there a way to identify just the failed scenario names instead of the entire feature? – Rewire

how will this help when we are trying to run our tests via maven? – Cryostat

This is no longer the best answer, use the retry flag and pass an integer. – Severally

If one gets parse error like me one needs to change IO.read('rerun.txt') with IO.read('rerun.txt').gsub(/\n/, ' ') in cucumber.yml – Demoss

Answer 2

Here is my simple and neat solution.

Step 1: Write your cucumber java file as mentioned below with rerun:target/rerun.txt. Cucumber writes the failed scenarios line numbers in rerun.txt as shown below.

features/MyScenaios.feature:25
features/MyScenaios.feature:45

Later we can use this file in Step 2. Name this file as MyScenarioTests.java. This is your main file to run your tagged scenarios. If your scenarios has failed test cases, MyScenarioTests.java will write/mark them rerun.txt under target directory.

@RunWith(Cucumber.class)
@CucumberOptions(
    monochrome = true,
    features = "classpath:features",
    plugin = {"pretty", "html:target/cucumber-reports",
              "json:target/cucumber.json",
              "rerun:target/rerun.txt"} //Creates a text file with failed scenarios
              ,tags = "@mytag"
           )
public class MyScenarioTests   {

}

Step 2: Create another scenario file as shown below. Let's say this as FailedScenarios.java. Whenever you notice any failed scenario run this file. This file will use target/rerun.txt as an input for running the scenarios.

@RunWith(Cucumber.class)
@CucumberOptions(
    monochrome = true,
    features = "@target/rerun.txt", //Cucumber picks the failed scenarios from this file 
    format = {"pretty", "html:target/site/cucumber-pretty",
            "json:target/cucumber.json"}
  )
public class FailedScenarios {

}

Everytime if you notice any failed scenarios run the file in Step 2

Annabel

I tried exact same steps as you mention above. I could see that rerun.txt has been generated but still failed scenario are not getting executed – Mcburney

You have to use FailedScenarios class to execute your tests. In other words, you need to right click on FailedScenarios and hit run. – Annabel

I could achieve it. I have posted my answer here. #49132947 – Mcburney

@vkrams This works for me when running from a IDE. But, when running from the command line with gradle, the @target/rerun.txt part isn't recognized and I get the Undefined scenarios message. Did you ever encounter something like this? – Strickler

@MateMrše This solution works for maven project. For gradle, make sure rerun.txt written in target directory if there is atleast one failed scenario. Also make sure you are using right cucumber version 1.2.5 – Annabel

will this solution not get into a infinite loop if the script fails due to some other issue? how is that handled here? have you controlled the number of reruns anywhere? – Cryostat

I tried exactly the same steps as above, using Eclipse. I can see the rerun.txt file, which says: Login.feature:4 Presumably that means the fourth step in that particular feature file. Unfortunately, when I run the failed scenario class, I get: java.lang.IllegalArgumentException: Neither found on file system or on classpath: Not a file or directory: C:\Users\ttoth-fejel\Git\TestAutomation\Cucumber\Login.feature, No resource found for: classpath:Login.feature But if the classpath didn't include that file, the first test never could run. How do I debug this? Or add the proper classpath? – Tailing

Answer 3

I know this is old, but I found my way here first and then later found a much more up to date answer (not the accepted, Cyril Duchon-Doris' answer): https://mcmap.net/q/153267/-cucumber-re-run-failed-scenarios-automatically-with-a-tag

Since cucumber 3.0 you can use --retry to specify how many times to retry a scenario that failed. https://cucumber.io/blog/open-source/announcing-cucumber-ruby-3-0-0/

Just tack it on to your cucumber command: cucumber ... --retry 2

Thirsty

Is it possible to implement it in @CucumberOptions in runner? I run tests through Maven – Delacroix

This does not work for cucumber-jvm, though. – Mutter

Answer 4

task cucumber() {
    dependsOn assemble, compileTestJava
    doLast {
        javaexec {
            main = "io.cucumber.core.cli.Main"
            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
            args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
                    '--plugin', "rerun:target/rerun.txt",
                    '--glue', 'steps',
                    'src/test/resources']
        }
    }
}

task cucumberRerunFailed() {
    doLast {
        javaexec {
            main = "io.cucumber.core.cli.Main"
            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
            args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
                    '@target/rerun.txt']
        }
    }
}

Sigman

Answer 5

You need at least version 1.2.0 in order to use the @target/rerun.txt new feature. After that just create a runner that runs at the end and uses this file. Also, if you are using Jenkins, you can put a tag on the random failures features so the build doesn't fails unless after being ran twice.

Booboo

Can you please explain this in detail? I am trying to run this via Jenkins/command line. – Singlestick