If I @Ignore a test class in JUnit4, does @BeforeClass still run?

Question

Quick background: I've been hunting down a Maven / Surefire test-running problem for days now, and I've narrowed it down to a small number suspect of tests. The behavior I'm seeing is insane. I start with mvn clean test: 250 tests run, 0 skipped. Now, I move the suspect test into src/test/java and try again: 146 tests run, 0 skipped! The output of Maven gives no clue that other tests aren't being run, even with the -X flag.

That brings me to my question: the reason I call the test 'suspect' is that the whole class is decorated with @Ignore, so I would imagine that including it in my test sources should have no effect at all. Then it occurred to me -- those classes have @BeforeClass/@AfterClass methods that manage a dummy Zookeeper server. It's resulted in wonky behavior before, which is why we have the tests @Ignored.

If JUnit is running the before/after code but ignoring the tests, I have no idea what might happen (but it'd probably be super bad). Is this happening? Is this supposed to happen? If so, how am I supposed to say "for reference, here's a test that should work but needs fixing" when it includes @BeforeClass / @AfterClass? Also of substantial interest: what the hell is this doing to Surefire / Maven, that it causes unrelated tests to fall off the face of the Earth?

Hygrothermograph

On my machine mvn clean test does not run @BeforeClass in an @Ignore class. The Skipped counter is increasing by only one regardless of the number of @Test methods in the @Ignore class. (Maven 2.2.1, jUnit 4.9) You should attach some code and version numbers. Are you using a custom test runner? – Fernery

I am not using a custom runner. It's Maven 3.0.something, JUnit 4.8.1. – Hygrothermograph

What's the version number of your surefire plugin? – Fernery

Answer 1

If you have a test with the @Ignore annotation, then it is normal behaviour for the @BeforeClass & @AfterClass to get run, whether or not all of the tests are @Ignored.

If, however, the Class has an @Ignore annotation, then the @BeforeClass & @AfterClass don't get run.

For maven, if you don't want to run any tests in a particular class, then you have to ignore them in surefire or failsafe. Add this to the maven configuration (see Maven Surefire Plugin)

<excludes>
 <exclude>**/FoobarTest.class</exclude>
</excludes>

Magical

Ugh. So that answers the "does it work this way" question -- though, WHAT THE HELL this is so stupid, why would I want the before and after code to run for a class whose tests aren't going to run? -- but it leaves the question of how Surefire is misbehaving so strangely. Why would some random, unrelated tests not run at all when this happens? – Hygrothermograph

Is it possible that you've overlooked that "the whole class is decorated with @Ignore"? – Fernery

He said that the tests were ignored, not the class. You're right that the class itself can be ignored though. – Magical

Um, but your point of view is right too. It also could mean that he annotated every method with @Ignore. @Coderer? – Fernery

I annotated just the class with @Ignore. As @Matthew points out, this means before/after will still run. I maintain that this is a terrible, terrible idea. – Hygrothermograph

OK just noticed the edits, now Matthew says the before/after won't run when the class is ignored? Maybe this varies by version of surefire / junit / maven? I can't verify whether they run or not, but I can verify that my test executions go all kinds of wonky if I leave the test class ignored rather than commented-out. – Hygrothermograph

Going to assume this was something buggy in my setup, as I'm having trouble reproducing. – Hygrothermograph

Answer 2

Environment: JDK 1.6, surefire plugin 2.9, jUnit 4.8.1, Maven 3.0, 3.0.3, 2.2.1.

I created this test class:

import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Ignore;
import org.junit.Test;

@Ignore
public class IgnoreTest {

    @BeforeClass
    public static void beforeClass() {
        System.out.println("BEFORE CLASS");
    }

    @AfterClass
    public static void afterClass() {
        System.out.println("AFTER CLASS");
    }

    @Test
    public void test1() throws Exception {
        System.out.println("test1");
    }

    @Test
    public void test2() throws Exception {
        System.out.println("test2");
    }

    @Test
    public void test3() throws Exception {
        System.out.println("test3");
    }
}

Then mvn clean test print this:

Running hu.palacsint.stackoverflow.q7535177.IgnoreTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.015 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

Works as you expected. If I remove the @Ignore and run mvn clean test again it prints this:

Running hu.palacsint.stackoverflow.q7535177.IgnoreTest
BEFORE CLASS
test2
test1
test3
AFTER CLASS
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045 sec

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

So, it works for me with three different Maven versions. No @BeforeClass/@AfterClass was run in @Ignored classes.

There is one (maybe more) situation when @BeforeClass/@AfterClass methods could run in an @Ignored test class. It's when your ignored class has a not ignored subclass:

import org.junit.Test;

public class IgnoreSubTest extends IgnoreTest {

    @Test
    public void test4() throws Exception {
        System.out.println("test4 subclass");
    }

}

Results of mvn clean test:

Running hu.palacsint.stackoverflow.q7535177.IgnoreTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.047 sec
Running hu.palacsint.stackoverflow.q7535177.IgnoreSubTest
BEFORE CLASS
test4 subclass
test1
test2
test3
AFTER CLASS
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec

Results :

Tests run: 5, Failures: 0, Errors: 0, Skipped: 1

In this case the @BeforeClass and the @AfterClass methods runs because they are methods of the IgnoreSubTest test class.

Fernery

Two things: one, note that your first execution shows "Tests run: 1" even though everything is ignored, and "Skipped: 1" even though there are three methods decorated with @Test. So, Surefire (or maybe JUnit?) is already hosed there. Two, I have way too much code to paste everything here but believe me when I say that @Ignoreing the class causes many unrelated tests to not run, while commenting out the @BeforeClass methods fixes the issue. I just wish I could explain why. – Hygrothermograph

jUnit said that "Tests run: 5 ... Skipped: 1" - It means to me that it tried to run 5 tests and 1 of the 5 has @Ignore. I think its correct. jUnit just works in this way. For the 2nd issue: you should narrow your investigation to only a few tests. Save/commit your code, pick just 3-4 relevant tests and delete the others and try to reproduce a bug with this small subset of your tests. Then edit your question :) – Fernery