How to get the exception that was thrown when a Cucumber test failed in Java?
Asked Answered
V

8

12

I can perform actions on test failure by using:

@After
public void afterTest(Scenario scenario) {
    if (scenario.isFailed()) {
        /*Do stuff*/
    }
}

However some of the actions I need to perform depend on the Exception that was thrown and in what context it was thrown. Is there a way to get the Throwable that caused the test to fail? For example in JUnit I would do this by extending TestWatcher and add a rule to my tests:

@Override
protected void failed(Throwable e, Description description) {
    /*Do stuff with e*/
}

However the cucumber-junit iplementation does not allow the use of rules, so this solution would not work with Cucumber.

I don't think I need to explain why accessing a thrown exception on test failure would be useful, however I will still provide an Example:

My test environment is not always stable, so my tests might fail unexpectedly at any moment (there's no specific place I can try to catch the exception since it could occur at any time). When this happens I need the test to reschedule for another attempt, and log the incident so that we can get some good statistical data on the environment instability (when, how frequent, how long etc.)

Vesuvianite answered 1/3, 2017 at 21:38 Comment(0)
P
12

The problem with the work around suggested by Frank Escobar:

By using reflection to reach into a frameworks internals you're depending on implementation details. This is a bad practice, when ever the framework changes its implementation your code may break as you will observe in Cucumber v5.0.0.

Hooks in Cucumber are designed to manipulate the test execution context before and after a scenario. They're not made to report on the test execution itself. Reporting is cross cutting concern and best managed by using the plugin system.

For example:

package com.example;

import io.cucumber.plugin.ConcurrentEventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;

public class MyTestListener implements ConcurrentEventListener {
    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseFinished.class, this::handleTestCaseFinished);
    }

    private void handleTestCaseFinished(TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        Result result = event.getResult();
        Status status = result.getStatus();
        Throwable error = result.getError();
        String scenarioName = testCase.getName();
        String id = "" + testCase.getUri() + testCase.getLine();
        System.out.println("Testcase " + id + " - " + status.name());
    }
}

When using JUnit 4 and TestNG you can activate this plugin using:

@CucumberOptions(plugin="com.example.MyTestListener")

With JUnit 5 you add it to junit-platform.properties:

cucumber.plugin=com.example.MyTestListener 

Or if you are using the CLI

--plugin com.example.MyTestListener 
Projective answered 14/3, 2020 at 11:36 Comment(0)
R
5

I've implemented this method using reflections. You can't access directly to steps errors (stack trace). I've created this static method which allows you to access to "stepResults" attribute and then you can iterate and get the error and do whatever you want.

import cucumber.runtime.ScenarioImpl; 
import gherkin.formatter.model.Result;  
import org.apache.commons.lang3.reflect.FieldUtils;  
import java.lang.reflect.Field;  
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) {
  if (scenario.isFailed())
    logError(scenario);
}


private static void logError(Scenario scenario) {
   Field field = FieldUtils.getField(((ScenarioImpl) scenario).getClass(), "stepResults", true);
   field.setAccessible(true);
   try {
       ArrayList<Result> results = (ArrayList<Result>) field.get(scenario);
       for (Result result : results) {
           if (result.getError() != null)
               LOGGER.error("Error Scenario: {}", scenario.getId(), result.getError());
       }
   } catch (Exception e) {
       LOGGER.error("Error while logging error", e);
   }
}
Romanaromanas answered 31/8, 2018 at 13:57 Comment(11)
Is there a way I can get this result.getError() in stack trace?Oberhausen
@Oberhausen result.getError().getStackTrace() this one? getError() returns ThrowableRomanaromanas
This approach no longer works with the later versions of Cucumber as ScenarioImpl no longer seems to be accessibile in cucumber.runtime. Any ideas?Chaco
@Chaco from what version specifically?Romanaromanas
4.8.0 - the latest in the io.cucumber maven repoChaco
@Chaco I've posted the solution for 4.8.0 recentlyRomanaromanas
Amazing! Don't suppose you have ever tried to do something similar with Cucumber.js?Chaco
@Chaco Also, I've added the example for cucumber-jsRomanaromanas
Thanks for the answer. Just adding the imports here in case anyone may be looking for them. Cheers. import cucumber.runtime.ScenarioImpl; import gherkin.formatter.model.Result; import org.apache.commons.lang3.reflect.FieldUtils; import java.lang.reflect.Field; import java.util.ArrayList;Methylnaphthalene
@FrancislainyCampos Thanks, I've added it too.Romanaromanas
Hi @FrankEscobar, sorry just wondering if you'd know how to transfer this to new cucumber 5 version? It stopped working for me this week when I updated. Many thanks!Methylnaphthalene
G
2

You can to this by writing your own custom implementation of Formatter & Reporter interface. The empty implementation of Formatter is the NullFormatter.java which you can extend. You will need to provide implementations for the Reporter interface.

The methods which would be of interest will be the result() of the Reporter interface and possibly the done() method of Formatter. The result() has the Result object which has the exceptions.

You can look at RerunFormatter.java for clarity.

Github Formatter source

public void result(Result result) {
      //Code to create logs or store to a database etc...
      result.getError();
      result.getErrorMessage();
}

You will need to add this class(com.myimpl.CustomFormRep) to the plugin option.

plugin={"pretty", "html:report", "json:reports.json","rerun:target/rerun.txt",com.myimpl.CustomFormRep}

More details on custom formatters.

You can use the rerun plugin to get a list of failed scenarios to run again. Not sure about scheduling a run of failed tests, code to create a batch job or schedule one on your CI tool.

Guiscard answered 2/3, 2017 at 7:1 Comment(0)
R
1

This is the workaround for cucumber-java version 4.8.0 using reflection.

import cucumber.api.Result;
import io.cucumber.core.api.Scenario;
import io.cucumber.core.logging.Logger;
import io.cucumber.core.logging.LoggerFactory;
import io.cucumber.java.After;
import org.apache.commons.lang3.ClassUtils;
import org.apache.commons.lang3.reflect.FieldUtils;

import java.io.IOException;
import java.lang.reflect.Field;
import java.net.URL;
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) throws IOException {
    if(!scenario.getStatus().isOk(true)){
        logError(scenario);
    }
}

private static void logError(Scenario scenario) {
    try {
        Class clasz = ClassUtils.getClass("cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor");
        Field fieldScenario = FieldUtils.getField(clasz, "scenario", true);
        fieldScenario.setAccessible(true);
        Object objectScenario =  fieldScenario.get(scenario);

        Field fieldStepResults = objectScenario.getClass().getDeclaredField("stepResults");
        fieldStepResults.setAccessible(true);

        ArrayList<Result> results = (ArrayList<Result>) fieldStepResults.get(objectScenario);
        for (Result result : results) {
            if (result.getError() != null) {
                LOGGER.error(String.format("Error Scenario: %s", scenario.getId()), result.getError());
            }
        }
    } catch (Exception e) {
        LOGGER.error("Error while logging error", e);
    }
}
Romanaromanas answered 22/10, 2019 at 14:33 Comment(3)
Above function is not working with cucumber-java version 5.2.0. It gives the exception of cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor class not found.Weightlessness
@Weightlessness In the description clearly I'm specifying is for cucumber-java version 4.8.0. When the library has a major change like from 4 to 5, it's because there is no compatibility. Maybe version 5 could implement this out of the box.Romanaromanas
Can you please guide for ver 5.2.0 as I am totally new and can't find any help regarding this.Weightlessness
I
0

If you just want to massage the result being sent to the report then you can extend the CucumberJSONFormatter and override the result method like this:

public class CustomReporter extends CucumberJSONFormatter {

    CustomReporter(Appendable out) {
        super(out);
    }

    /**
     * Truncate the error in the output to the testresult.json file.
     * @param result the error result
    */
    @Override
    void result(Result result) {
        String errorMessage = null;
        if (result.error) {
            errorMessage = "Error: " + truncateError(result.error);
        }
        Result myResult = new Result(result.status, result.duration, errorMessage);
        // Log the truncated error to the JSON report
        super.result(myResult);
    }
}

Then set the plugin option to:

plugin = ["com.myimpl.CustomReporter:build/reports/json/testresult.json"]
Irresoluble answered 5/6, 2018 at 3:43 Comment(0)
R
0

For cucumber-js https://www.npmjs.com/package/cucumber/v/6.0.3

import { After } from 'cucumber'

After(async function(scenario: any) {
    const exception = scenario.result.exception
    if (exception) {
        this.logger.log({ level: 'error', message: '-----------StackTrace-----------' })
        this.logger.log({ level: 'error', message: exception.stack })
        this.logger.log({ level: 'error', message: '-----------End-StackTrace-----------' })
    }
})

enter image description here

Romanaromanas answered 23/10, 2019 at 11:12 Comment(5)
This doesn't seem to work because exception does not exist in the scenario object with the latest version of cucumber. Any other ideas? Perhaps with reflection?Chaco
@Chaco To said that you have to be sure. That exception attribute is Error type.Romanaromanas
I am using latest version of cucumber via npm and don't get this. I just see duration and status.Chaco
Just an update, turns out it's the "Scenario: any" as param type that is key here. It works with any set as type. If I used HookScenarioResult as type it can't find exception.Chaco
@Chaco that's right. With HookScenarioResult you are parsing the scenario to that object and that object doesn't have the exception.Romanaromanas
S
0

After a lot of experimentation I now removed the Before/After-Annotations and rely on Cucumber-Events instead. They contain the TestCase (which is what the Scenario-class wraps) and a Result where you can call getError(); to get the Throwable.

Here is a simple example to get it working

import io.cucumber.plugin.EventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;
import io.cucumber.plugin.event.TestCaseStarted;
import org.openqa.selenium.WebDriver;

public class TestCaseListener implements EventListener {
    @Override
    public void setEventPublisher(final EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseStarted.class, this::onTestCaseStarted);
        publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished);
    }

    public void onTestCaseStarted(TestCaseStarted event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Starting " + testCase.getName());

        // Other stuff you did in your @Before-Method.
        // ...
    }

    private void onTestCaseFinished(final TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Finished " + testCase.getName());

        Result result = event.getResult();
        if (result.getStatus() == Status.FAILED) {
            final Throwable error = result.getError();
            error.printStackTrace();
        }

        // Other stuff you did in your @After-Method.
        // ...
    }
}

All that's left to do is to register this class as a Cucumber-Plugin. I did this by modifying my @CucumberOptions-annotation:

@CucumberOptions(plugin = {"com.example.TestCaseListener"})

I find this much cleaner than all of this reflection-madness, however it requires a lot more code-changes.

Edit

I don't know why, but this caused a lot of tests to randomly fail in a multithreaded environment. I tried to figure it out, but now also use the ugly reflections mentioned in this thread:

public class SeleniumUtils {
private static final Logger log = LoggerFactory.getLogger(SeleniumUtils.class);

private static final Field field = FieldUtils.getField(Scenario.class, "delegate", true);
private static Method getError;

public static Throwable getError(Scenario scenario) {
    try {
        final TestCaseState testCase = (TestCaseState) field.get(scenario);
        if (getError == null) {
            getError = MethodUtils.getMatchingMethod(testCase.getClass(), "getError");
            getError.setAccessible(true);
        }
        return (Throwable) getError.invoke(testCase);
    } catch (Exception e) {
        log.warn("error receiving exception", e);
    }
    return null;
}
}
Stieglitz answered 14/2, 2020 at 11:49 Comment(1)
Rather then using EventListener you should use ConcurrentEventListener. Otherwise Cucumber will only provide you the events after the concurrent execution is over.Projective
E
0

It took a bit of research and trial, but I found the solution. No need to add any plugin or listeners. Just add this in your hooks class and you are good to go.

@AfterStep
public void getErrors( Scenario scenario ) throws NoSuchFieldException, IllegalAccessException
{
    Field delegate = scenario.getClass().getDeclaredField("delegate");
    delegate.setAccessible(true);

    TestCaseState tcs = ( TestCaseState ) delegate.get(scenario);

    Field stepResults = tcs.getClass().getDeclaredField("stepResults");
    stepResults.setAccessible(true);

    ArrayList<Result> results = ( ArrayList<Result> ) stepResults.get(tcs);
    for ( Result result : results )
    {
        if ( result.getError() != null )
        {
            System.out.println(result.getError());
            //Pass it to extent report or do anything with it
        }
    }
}
Earplug answered 31/8, 2023 at 7:56 Comment(0)

© 2022 - 2025 — McMap. All rights reserved.