How to run all JUnit tests in a category/suite with Ant?

Question

I'm using JUnit Categories and ClassPathSuite in a setup similar to that described in this answer. To recap:

public interface FastTests {
}

@RunWith(Categories.class)
@Categories.IncludeCategory(FastTests.class)
@Suite.SuiteClasses(AllTests.class)
public class FastTestSuite {
}

@RunWith(ClasspathSuite.class) 
public class AllTests {
}

...where AllTests makes use of the ClasspathSuite library.

A test class that's part of the FastTests category would look like this:

@Category(FastTests.class)
public class StringUtilsTest {
    //  ...
}

When I run "FastTestSuite" in my IDE, all tests with the FastTests annotation are executed, nice & smooth:

Now, I want to do the same thing with Ant. (To my surprise, I couldn't easily find instructions for this on SO.) In other words, I need an Ant target that runs all tests with the FastTests annotation.

I've tried some simplistic approaches using <test> or <batchtest>...

 <junit showoutput="true" printsummary="yes">
     <test name="fi.foobar.FastTestSuite"/>
     <formatter type="xml"/>
     <classpath refid="test.classpath"/>
 </junit>

... but no luck, so far.

Edit: Besides the IDE, it works fine with JUnitCore on the command line:

$ java -classpath "classes:WebContent/WEB-INF/lib/*" org.junit.runner.JUnitCore fi.foobar.FastTestSuite
.............
Time: 0.189

OK (13 tests)

Nabala

is name="..." a path like org/test/TestSuite or in the package annotation org.test.TestSuite? What is the error? – Remediable

@oers: Latter; fully qualified classname. No real error message, it just... fails. unittest-fast: [junit] Running fi.foobar.FastTestSuite [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.053 sec [junit] Test fi.foobar.FastTestSuite FAILED – Nabala

change the formatter to plain and change printsummary to withOutAndErr . That should show the reason for the errors. I think there is an exception thrown in there. Your setup is definitely the right way to execute Suites. – Remediable

@oers, thanks! Actually the error was there even without "withOutAnderr" (I forgot I need to look at the separate report files): Testcase: initializationError took 0.002 sec Caused an ERROR null at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) Btw, as I edited on the question, this works with JUnitCore, which seems encouraging... – Nabala

Hi. I have a question that is related. feel free to chime in: #15777218 – Nonagon

Answer 1

Right, I got it working with <batchtest> quite simply:

<junit showoutput="true" printsummary="yes" fork="yes">
    <formatter type="xml"/>
    <classpath refid="test.classpath"/>
    <batchtest todir="${test.reports}">
        <fileset dir="${classes}">
            <include name="**/FastTestSuite.class"/>
        </fileset>
    </batchtest>
</junit>

I had tried <batchtest> earlier, but had made the silly mistake of using "**/FastTestSuite.java" instead of "**/FastTestSuite.class" in the <include> element... Sorry about that :-)

NB: it's necessary to set fork="yes" (i.e., run the tests in a separate VM); otherwise this will also produce "initializationError" at java.lang.reflect.Constructor.newInstance(Constructor.java:513) like <test> (see comments on the question). However, I couldn't get <test> working even with fork="yes".

The only shortcoming is that this produces just one JUnit XML report file (TEST-fi.foobar.FastTestSuite.xml) which makes it look like all the (hundreds) of tests are in one class (FastTestSuite). If anyone knows how to tweak this to show the tests in their original classes & packages, please let me know.

Nabala

Using the junit4 Ant Task bundled inside the randomized testing project from carrotsearch: [labs.carrotsearch.com/randomizedtesting.html] I am able to run tests using Categories and ClassPathSuite - but not required to be inside a <batchtest>. Their JSON-type report generates output reports that have the tests broken down by their original packages - and look pretty nice, too. – Rand

Is there a way of doing this by specifying the category name (class) itself in the build.xml? Like in maven where you can say 'mvn test -Dgroups=test.classpath.FastTests' – Brickle

A workaround for your shortcoming is to apply a xslt before junitreport. You just need to split the initial xml in smaller ones – Letendre

Added another answer which fixes your 'shortcomming' because it will produce separate JUnit XML report files. – Pasteurism

Answer 2

I found a workaround to use ant's junit task to run test of a specific Category:

<target name="test" description="Run Selenium tests by category">
    <fail unless="category.name" message="Please specify 'category.name' property"/>

    <junit showoutput="true" printsummary="true" fork="true">
        <formatter type="xml"/>
        <classpath refid="test.classpath"/>

        <batchtest todir="${test.reports}" haltonfailure="false">
            <fileset dir="${classes}">
                <!-- regex '^\s*@Category' part added to ensure @Category annotation is not commented out -->
                <containsregexp expression="^\s*@Category.*${category.name}"/>
            </fileset>
        </batchtest>
    </junit>
</target>

Execute test by supplying category.name property to ant like this:

ant -Dcategory.name=FastTests test

Using batchtest will also produce separate JUnit XML report files per test (e.g. TEST-fi.foobar.FastTestClassN.xml).

Pasteurism

I am not able to reproduce this because the containsregexp appears to be searching the class file for the text of the category. However, the text is not in the binary file. – Thalia

This filtering using containsregexp has to be done before compilation. The provided solution won't work because containsregexp can't search inside .class files. So use this filter in your compilation ant task and by that way we can solve the problem. – Eloquence