TestNG skips test after raising Exception in @DataProvider method

Question

I'm a bit confused.

How can I get TestNG to report an error for a test?

// ...
@DataProvider(name = "foo")
public Object[][] provideData () {
    throw new SomeRuntimeException("Some error occurred. The test configuration "
            + "is somehow incorrect.");
}

This will just lead to test skipping. The exception doesn't even get logged. Moving this to a constructor will just get the exception logged but that's not enough...

I want a big fat error message.

At the moment, using a dedicated (self) test method does the job which at leasts shows some test failure...

Anyway, it would be nice to know how testNG's definition of an error looks like.

Thank you for any hints!

Kaz

Can you show us the code that calls this? Are you sure it's not somehow wrapped in a try/catch? – Whirlybird

The method referring to the dataProvider is not aware of any exception in any way. – Kaz

As far as I understand it the test method itself doesen*t call the dataprovider. A test method gets called with params provided by the Dataprovider. – Kaz

The DataProvider is part of the preamble for tests, and, as such, does not get reported as an error. It is your responsibility to catch and log any errors in the DataProvider method. – Septimal

Too bad. I wonder if there is any possibility to trigger an error directly. It does not need to be done in a DataProvider. I mean generally, is there a way? – Kaz

The behaviour of TestNG's DataProvider is really counterintuitive. I've just wasted a couple of hours on this issue: I couldn't understand why my tests weren't running. However, your question and Roman's answer put me on the right track: it turns out an exception was being thrown from within my DataProvider. Thanks to you both! – Trichina

Answer 1

Here is the article which proposes fair set of alternatives with detailed enough explanations:

Fail instead of Skip a Test when TestNG’s DataProvider throws an Exception

For me best way happened to add test for data providers and here is brief illustration of idea:

public class MyClass {

    @DataProvider(name = "my-data-provider")
    private Object [][] myProvider() {
        // ...
    }

    @Test
    public void testDataProviders() {
        Assert.assertTrue(myProvider().length > 0);
    }

    @Test
    // ... Real tests.

}

Danieladaniele

The question is pretty old but the problem still is highly topical in my eyes. Thank you very much! The article is really helpful. – Kaz

Answer 2

Today, the most recent version of TestNG allows the user to explicitly tell the TestNG runner to treat DataProvider exceptions as Failures.

If you're running TestNG programmatically, this can be done like so -

TestNG tng = new TestNG();
tng.propagateDataProviderFailureAsTestFailure();

If you're running TestNG conventionally, you can also set the attribute on the DataProvider annotation like so -

@DataProvider(name = "DataProviderName", propagateFailureAsTestFailure = true)
    public Object[][] dataProviderMethod(){
        return new Object[][]{{"row1-column1", "row1-column2"},
                {"row2-column1", "row2-column2"}};
    }

Reference:

https://github.com/testng-team/testng/issues/217

Cadet