Does new JUnit 4.8.1 @Category render test suites almost obsolete?

Question

Given question 'How to run all tests belonging to a certain Category?' and the answer would the following approach be better for test organization?

1. define master test suite that contains all tests (e.g. using ClasspathSuite)
2. design sufficient set of JUnit categories (sufficient means that every desirable collection of tests is identifiable by one or more categories)
3. qualify each test with relevant category(ies)
4. define targeted test suites based on master test suite and set of categories

Example:

• identify categories for speed (slow, fast), dependencies (mock, database, integration, etc.), function (fill in applicable categories), domain (fill in applicable categories), etc.
• demand that each test is properly qualified (tagged) with relevant set of categories.
• create master test suite using ClasspathSuite (all tests found in classpath)
• create targeted suites by qualifying master test suite with categories, e.g. mock test suite, fast database test suite, slow integration for domain X test suite, etc.

My question is more like soliciting approval rate for such approach vs. classic test suite approach. One unbeatable benefit is that every new test is immediately contained by relevant suites with no suite maintenance. One concern is proper categorization of each test.

Exordium

added blog entry: novyden.blogspot.com/2011/06/… – Exordium

I would say yes. But then I never really understood traditional JUnit suites anyway. :) Such hassle maintaining category info in a separate place! If it wasn't for the new @Categories & ClasspathSuite, I'd still prefer simple naming patterns. +1 for raising a good question. – Feudist

@Feudist - thanks. Naming pattern approach is definitely a hack with respect to tests. And as such it's not very flexible as it doesn't allow to mix and match tests into different suites based on multiple categories. I also like to think of test name reflecting its purpose - not its properties. – Exordium

Answer 1

One of the trade-off you have to consider is IMHO that categories are defined within the tests and suites are defined outside. This means, that when you use suites, you know which tests will be run, but when using categories, you do not know unless you inspect tests or simple run them. At the same time, when you look at the test using test suites you do not know in which suite it is contained, unless checking your suites, but using categories, you see it immidiatly.

Selfrising

valid point. It depends on how important one way of inspecting tests or the other for a team. I can expect that with embracing categories priorities shift. IMHO categories fit annotation/declarative unit testing better. – Exordium

Problem with categories, although I also like them, is that they are one meta level higher than tests and therefore you lose some information. To present that point, imagine having many categories and a typo - you won't find it (AFAIK). With Suites you have type checking to ensure you used valid names. I would say, that this is an issue which can be generalized into JUnit3 vs JUnit4 discussion. 4 is more powerful, but is further from the actual language (because annotations are just metadata, not metaprogramming) and 3 although not so flexible, it is (relativly) just another Java code. – Transaction

annotations are metadata turned declarative programming in many cases... I agree one always has to strike proper balance when using them. If you think of this test suites are one level higher than tests too: they don't bring new tests - they just organize them based on goals and priorities. That should be the reason why JUnit 4 moved suites to annotation proper. – Exordium

my point is - suites are within the language and object model, so existing language tool can offer much support, but annotations are outside, there can not be any general support for them. I would say the preference is also about your workflow. If you have some standardized categories as you mentioned in you question, than annotations are really good, because you can specify the categories withing the test as you writte it. – Transaction

I recently stumbled upon a good read on the topic on DZone: <a href="java.dzone.com/articles/… Tests Using JUnit Categories</a> – Transaction

@Gabriel Ščerbák: Note that Categories in JUnit are typechecked. The value provided to the annotation Category must be a class (or interface) literal, and is therefore subject to typechecking. – Emmaline

the correct link of Gabriel's dzone article: dzone.com/articles/grouping-tests-using-junit – Strohbehn