How to run all tests belonging to a certain Category in JUnit 4
Asked Answered
K

6

73

JUnit 4.8 contains a nice new feature called "Categories" that allows you to group certain kinds of tests together. This is very useful, e.g. to have separate test runs for slow and fast tests. I know the stuff mentioned in JUnit 4.8 release notes, but would like to know how I can actually run all the tests annotated with certain category.

The JUnit 4.8 release notes show an example suite definition, where SuiteClasses annotation selects the tests from certain category to run, like this:

@RunWith(Categories.class)
@IncludeCategory(SlowTests.class)
@SuiteClasses( { A.class, B.class }) // Note that Categories is a kind of Suite
public class SlowTestSuite {
  // Will run A.b and B.c, but not A.a
}

Does anyone know how I could run all the tests in SlowTests category? It seems that you must have the SuiteClasses annotation...

Keldon answered 1/2, 2010 at 12:29 Comment(2)
Hi. I have a question that is related. feel free to chime in: #15777218Octa
It is not directly related, but this is a way to compute 'Tests by Category' counter.Samiel
K
65

I found out one possible way to achieve what I want, but I don't consider this to be the best possible solution as it relies on ClassPathSuite library that is not part of JUnit.

I define the test suite for slow tests like this:

@RunWith(Categories.class)
@Categories.IncludeCategory(SlowTests.class)
@Suite.SuiteClasses( { AllTests.class })
public class SlowTestSuite {
}

AllTests class is defined like this:

@RunWith(ClasspathSuite.class)
public class AllTests {
}

I had to use ClassPathSuite class from ClassPathSuite project here. It will find all the classes with tests.

Keldon answered 1/2, 2010 at 13:4 Comment(4)
It's actually a quite reasonable sollution. Thanks for anwesering your own question since it's a really good one :-)Clareclarence
For anyone wondering how to automate running a category of tests (with this exact setup) using Ant, this question might be useful.Hasdrubal
As a compliment to my question https://mcmap.net/q/275741/-does-new-junit-4-8-1-category-render-test-suites-almost-obsolete/59470 and detailed explanation I added a blog entry: novyden.blogspot.com/2011/06/…Foredate
It is not directly related, but this is a way to compute 'Tests by Category' counter.Samiel
B
7

Here are some of the main differences between TestNG and JUnit when it comes to groups (or categories, like JUnit calls them):

  • JUnit's are typed (annotations) while TestNG's are strings. I made this choice because I wanted to be able to use regular expressions when running tests, for example "run all the tests that belong to the group "database*". Also, having to create a new annotation whenever you need to create a new category is annoying, although it has the benefit that an IDE will tell you right away where this category is used (TestNG shows you this in its reports).

  • TestNG separates very clearly your static model (the code of your tests) from the runtime model (which tests get run). If you want to run the groups "front-end" first and then "servlets", you can do this without having to recompile anything. Because JUnit defines groups in annotations and you need to specify these categories as parameters to the runner, you usually have to recompile your code whenever you want to run a different set of categories, which defeats the purpose in my opinion.

Belt answered 13/6, 2011 at 1:28 Comment(1)
We built our own categories support into our JUnit tests in a very similar way to JUnit, the main difference being that instead of the @Categories.IncludeCategory annotation, we made ours configurable via a system property. Why this was too hard for JUnit to do for us is anybody's guess.Fluorescein
L
6

One downside to Kaitsu's solution is that Eclipse will run your tests twice, and the SlowTests 3 times, when running all the tests in a project. This is because the Eclipse will run all the tests, then the AllTests suite, then the SlowTestSuite.

Here is a solution that involves creating subclasses of the Kaitsu solution test runners to skip the suites unless a certain system property is set. A shameful hack, but all I have come up with so far.

@RunWith(DevFilterClasspathSuite.class)
public class AllTests {}

.

@RunWith(DevFilterCategories.class)
@ExcludeCategory(SlowTest.class)
@SuiteClasses(AllTests.class)
public class FastTestSuite
{
}

.

public class DevFilterCategories extends Suite
{
    private static final Logger logger = Logger
        .getLogger(DevFilterCategories.class.getName());
    public DevFilterCategories(Class<?> suiteClass, RunnerBuilder builder) throws InitializationError {
        super(suiteClass, builder);
        try {
            filter(new CategoryFilter(getIncludedCategory(suiteClass),
                    getExcludedCategory(suiteClass)));
            filter(new DevFilter());
        } catch (NoTestsRemainException e) {
            logger.info("skipped all tests");
        }
        assertNoCategorizedDescendentsOfUncategorizeableParents(getDescription());
    }

    private Class<?> getIncludedCategory(Class<?> klass) {
        IncludeCategory annotation= klass.getAnnotation(IncludeCategory.class);
        return annotation == null ? null : annotation.value();
    }

    private Class<?> getExcludedCategory(Class<?> klass) {
        ExcludeCategory annotation= klass.getAnnotation(ExcludeCategory.class);
        return annotation == null ? null : annotation.value();
    }

    private void assertNoCategorizedDescendentsOfUncategorizeableParents(Description description) throws InitializationError {
        if (!canHaveCategorizedChildren(description))
            assertNoDescendantsHaveCategoryAnnotations(description);
        for (Description each : description.getChildren())
            assertNoCategorizedDescendentsOfUncategorizeableParents(each);
    }

    private void assertNoDescendantsHaveCategoryAnnotations(Description description) throws InitializationError {           
        for (Description each : description.getChildren()) {
            if (each.getAnnotation(Category.class) != null)
                throw new InitializationError("Category annotations on Parameterized classes are not supported on individual methods.");
            assertNoDescendantsHaveCategoryAnnotations(each);
        }
    }

    // If children have names like [0], our current magical category code can't determine their
    // parentage.
    private static boolean canHaveCategorizedChildren(Description description) {
        for (Description each : description.getChildren())
            if (each.getTestClass() == null)
                return false;
        return true;
    }
}

.

public class DevFilterClasspathSuite extends ClasspathSuite
{
    private static final Logger logger = Logger
        .getLogger(DevFilterClasspathSuite.class.getName());
    public DevFilterClasspathSuite(Class<?> suiteClass, RunnerBuilder builder) 
        throws InitializationError {
        super(suiteClass, builder);
        try
        {
            filter(new DevFilter());
        } catch (NoTestsRemainException e)
        {
            logger.info("skipped all tests");
        }
    }
}

.

public class DevFilter extends Filter
{
    private static final String RUN_DEV_UNIT_TESTS = "run.dev.unit.tests";

    @Override
    public boolean shouldRun(Description description)
    {
        return Boolean.getBoolean(RUN_DEV_UNIT_TESTS);
    }

    @Override
    public String describe()
    {
        return "filter if "+RUN_DEV_UNIT_TESTS+" system property not present";
    }
}

So, in your FastTestSuite launcher, just add -Drun.dev.unit.tests=true to the VM arguments. (Note that this solution references a fast test suite instead of a slow one.)

Leven answered 11/1, 2013 at 23:5 Comment(0)
F
2

To run categorized tests without specifying all of them explicily in @Suite.SuiteClasses annotation you can provide your own implementation of Suite. For example a org.junit.runners.ParentRunner can be extended. Instead of using an array of classes provided by @Suite.SuiteClasses, new implementation should perform search for categorized tests in classpath.

See this project as an example of such approach. Usage:

@Categories(categoryClasses = {IntegrationTest.class, SlowTest.class})
@BasePackage(name = "some.package")
@RunWith(CategorizedSuite.class)
public class CategorizedSuiteWithSpecifiedPackage {

}
Frequentation answered 9/11, 2012 at 20:27 Comment(0)
P
1

I am not sure, what exactly your problem is.

Just add all the tests to a suite (or hirachy of suites). Then use the Categories Runner and Include/ExcludeCategory annotation, to specify the categories you want to run.

A good idea might be to have one suite containing all the tests, and a couple of seperate suites referring to the first one, specifying the different set of Categories you neeed.

Penult answered 1/2, 2010 at 12:43 Comment(1)
My problem is that I have thousands of tests and I don't want to manually add them to any suites. I just want that tests with certain category are run. It shouldn't be so hard for JUnit to find out which tests have certain annotation as it actually does that anyway when finding test methods.Keldon
M
0

Not a direct answer to your problem, but maybe the general approach could be improved...

Why are your tests slow? Maybe the set-up lasts long (database, I/O etc.), maybe the tests are testing too much? If this is the case I would seperate the real unit-tests from the "long-running" ones, which often indeed are integration tests.

In my setups I have staging env, where unit-tests are run often and integration-tests constantly but more rarely (e.g. after each commit in version control). I have never worked with grouping for unit tests, because they should be loosely coupled alltogether. I only work with grouping and relationship of test-cases in integration-test setups (but with TestNG).

But good to know that JUnit 4.8 introduced some grouping features.

Metalwork answered 1/2, 2010 at 20:1 Comment(1)
Thanks Manuel for your comments! I don't really need to separate unit tests, but I use JUnit also for integration tests and want to separate them from unit tests. I've looked also at TestNG and it seems to make testing (and not just unit testing) nicer than JUnit. And it also has better documentation and a good book.Keldon

© 2022 - 2024 — McMap. All rights reserved.