Why does JUnit run test cases for Theory only until the first failure?

Question

Recently a new concept of Theories was added to JUnit (since v4.4).

In a nutshell, you can mark your test method with @Theory annotation (instead of @Test), make your test method parametrized and declare an array of parameters, marked with @DataPoints annotation somewhere in the same class.

JUnit will sequentially run your parametrized test method passing parameters retrieved from @DataPoints one after another. But only until the first such invocation fails (due to any reason).

The concept seems to be very similar to @DataProviders from TestNG, but when we use data providers, all the scenarios are run inspite of their execution results. And it's useful because you can see how many scenarious work/don't work and you can fix your program more effectively.

So, I wonder what's the reason not to execute @Theory-marked method for every @DataPoint? (It appears not so difficult to inherit from Theories runner and make a custom runner which will ignore failures but why don't we have such behaviour out of the box?)

UPD: I have created a fault-tolerant version of Theories runner and made it available for a public access: https://github.com/rgorodischer/fault-tolerant-theories

In order to compare it with the standard Theories runner run StandardTheoriesBehaviorDemo then FaultTolerantTheoriesBehaviorDemo which are placed under src/test/... folder.

Lowerclassman

Answer 1

Reporting multiple failures in a single test is generally a sign that the test does too much, compared to what a unit test ought to do. Usually this means either that the test is really a functional/acceptance/customer test or, if it is a unit test, then it is too big a unit test.

JUnit is designed to work best with a number of small tests. It executes each test within a separate instance of the test class. It reports failure on each test. Shared setup code is most natural when sharing between tests. This is a design decision that permeates JUnit, and when you decide to report multiple failures per test, you begin to fight against JUnit. This is not recommended.

Long tests are a design smell and indicate the likelihood of a design problem. Kent Beck is fond of saying in this case that "there is an opportunity to learn something about your design." We would like to see a pattern language develop around these problems, but it has not yet been written down. Source: http://junit.sourceforge.net/doc/faq/faq.htm#tests_12

To ignore assertion failures you can also use a JUnit error collector rule:

The ErrorCollector rule allows execution of a test to continue after the first problem is found (for example, to collect all the incorrect rows in a table, and report them all at once)

For example you can write a test like this.

public static class UsesErrorCollectorTwice {
  @Rule
  public ErrorCollector collector= new ErrorCollector();

  @Test
  public void example() {
    String x = [..]
    collector.checkThat(x, not(containsString("a")));
    collector.checkThat(y, containsString("b"));             
  }
}

The error collector uses hamcrest Matchers. Depending on your preferences this is positive or not.

Carpetbag

Answer 2

AFAIK, the idea is the same as with asserts, the first failure stops the test. This is the difference between Parameterized & Theories.

Parameterized takes a set of data points and runs a set of test methods with each of them. Theories does the same, but fails when the first assert fails.

Try looking at Parameterized. Maybe it provides what you want.

Ipsambul

see the upd (I thought you may find it interesting). – Lowerclassman

Answer 3

A Theory is wrong if a single test in it is wrong, according to the definition of a Theory. If your test cases don't follow this rule, it would be wrong to call them a "Theory".

Ruvolo

IMHO, for engineering purposes its not very useful. After running tests you want to know as much about correctness of your system as possible. Again, I don't see anything bad in knowing, in which and in how many cases the theory is wrong. Its much better than knowing just that the theory is wrong without any details. – Lowerclassman

It is supposed that you will write small test units, each of which is testing your application from one only side. Sometimes this check is better realized as a theory. It cannot be of low use for any purpose, because it is based on the base principle of practical logic. If it seems to you that it is bad for use in some subject in 99% it means that you have some problems in understanding the subject. – Ruvolo

"...engineering purposes its not very useful. After running tests you want to know as much about correctness of your system as possible...." +1 and I've rather switched to @Parametrized. – Amor