Conditionally ignoring tests in JUnit 4

Question

OK, so the @Ignore annotation is good for marking that a test case shouldn't be run.

However, sometimes I want to ignore a test based on runtime information. An example might be if I have a concurrency test that needs to be run on a machine with a certain number of cores. If this test were run on a uniprocessor machine, I don't think it would be correct to just pass the test (since it hasn't been run), and it certainly wouldn't be right to fail the test and break the build.

So I want to be able to ignore tests at runtime, as this seems like the right outcome (since the test framework will allow the build to pass but record that the tests weren't run). I'm fairly sure that the annotation won't give me this flexibility, and suspect that I'll need to manually create the test suite for the class in question. However, the documentation doesn't mention anything about this and looking through the API it's also not clear how this would be done programmatically (i.e. how do I programatically create an instance of Test or similar that is equivalent to that created by the @Ignore annotation?).

If anyone has done something similar in the past, or has a bright idea of how else I could go about this, I'd be happy to hear about it.

Sherrard

Answer 1

The JUnit way is to do this at run-time is org.junit.Assume.

 @Before
 public void beforeMethod() {
     org.junit.Assume.assumeTrue(someCondition());
     // rest of setup.
 }

You can do it in a @Before method or in the test itself, but not in an @After method. If you do it in the test itself, your @Before method will get run. You can also do it within @BeforeClass to prevent class initialization.

An assumption failure causes the test to be ignored.

Edit: To compare with the @RunIf annotation from junit-ext, their sample code would look like this:

@Test
public void calculateTotalSalary() {
    assumeThat(Database.connect(), is(notNull()));
    //test code below.
}

Not to mention that it is much easier to capture and use the connection from the Database.connect() method this way.

Strawberry

@notnoop, that isn't my observation at all. They are ignored. The IDEA test runner reports them that way, and a look at the JUnit source code shows that it reports the test as ignored. – Strawberry

To quote: "In the future, this may change, and a failed assumption may lead to the test being ignored." It in fact changed, as of 4.5 I believe. The current javadoc says: "The default JUnit runner treats tests with failing assumptions as ignored. Custom runners may behave differently." github.com/KentBeck/junit/blob/… – Strawberry

Thanks, that's exactly what I'm looking for. The semantics of Assume sound like exactly what I'm trying to do here, so even if a runner doesn't treat it as @Ignore I trust it will be handled appropriately. – Sherrard

Eclipse 3.6 with Junit 4.8.1 reports false Assumptions as a passing test. Same with ant 1.8.1. – Dree

@fijaaron, Regarding ANT, this is a known limitation of its XML format, and since so many tools rely on this format, changing it is a big deal, so it won't happen so fast. Regarding eclipse, since IDEA does it, I have no idea why eclipse hasn't caught up. Regarding the ANT functionality, see here: issues.apache.org/bugzilla/show_bug.cgi?id=43969 – Strawberry

Note that if you are using maven's surefire plugin with JUnit in this fashion, you need to use surefire 2.7.2 or greater to get tests halted because of Assume to show up as Skipped rather than success. – Parkins

That Eclipse reports failed assumptions as passing is a bug: bugs.eclipse.org/bugs/show_bug.cgi?id=359944 – Stateroom

If you have an After method though, it appears the After method still runs, but it doesnt look like Assume can be used in an After method. Is there anyway to prevent both the before and after methods from running? – Krispin

@JeffStorey, the after method only runs if the assume is in the method itself, not in before. That makes sense - if there was a before, there would be an after. You can gain more fine grained control using a TestRule. kentbeck.github.com/junit/javadoc/latest/org/junit/rules/… – Strawberry

@Yishai, I set up a simple test where I put Assume.assumeTrue(false) in the before method, but the after method still runs. It sounds like that is the opposite of what you expected... – Krispin

@JeffStorey, I set up a test as well, and it worked as I described, or at least I thought it did. I tried again, and you are correct. I guess TestRule is the way to go, then. – Strawberry

@Yishai, thanks. I'm not sure if TestRule will work in this case though. Trying to find a way to basically ignore the whole test class if the assumption is false without annotating every method. I could always write a custom runner if needed to process some annotation. – Krispin

@JeffStorey, then you are looking for a couple of things. One is the @BeforeClass annotation, where you can have your assumption fail there, which will skip the whole class. Another is @ClassRule (for the fine grained control, but over the whole class, one time). – Strawberry

Thanks, I never really thought about using Assume in a beforeClass method. I'll give that a shot. And I'll look into ClassRule as well. Thanks. – Krispin

That is AWESOME! I'm going to use this method to separate my smoke tests with both eclipse and maven. Sweet. – Scapegoat

This is not the correct answer. This will not cause the test to be ignored or "passed". It still runs the test method's code. The correct way is to use @KyleShrader's suggestion – Tyratyrannical

In my case I go for the Assume way. Sometimes the condition is different depending on the test, I may run test A with input of type A and run testB with input of type B; then @Before is not the way to go because it executes before every test, thus the condition must be the same. – Josefjosefa

N.B. the @After methods will still run, even if the failed assumption was in a @Before method. – Titanomachy

On Android, if I use this with @BeforeClass, at the end of test execution I get InstrumentationResultParser: test run failed: 'Test run failed to complete. Expected 204 tests, received 149'. So it's failing the entire test run because of the tests I'm intentionally skipping in @BeforeClass. Is there any way around this? – Ablate

Answer 2

You should checkout Junit-ext project. They have RunIf annotation that performs conditional tests, like:

@Test
@RunIf(DatabaseIsConnected.class)
public void calculateTotalSalary() {
    //your code there
}

class DatabaseIsConnected implements Checker {
   public boolean satisify() {
        return Database.connect() != null;
   }
}

[Code sample taken from their tutorial]

Ferbam

Thanks for this answer - an interesting alternative syntax for the functionality, though I'll be going with Assume directly so as not to introduce another dependency. – Sherrard

I personally prefer this solution. If you have many tests that should be run based on the same conditions, this would be far more ideal than having to use Assume in every test. Also, if this can be used on a class level rather than the method level, it will be even more ideal. – Ealing

I would prefer it, 'cuz this helps to run the test conditionally at run time. It suits where a number of unit test are going to run and the requirement is to run the unit tests on particular checker. I really amazed to see that junit-ext is not available on maven repository. How would we get avail this in maven project. – Catchascatchcan

An annotation like @RunIf separates the condition when a test should run from the actual test code, which I think is good. What I don't like is that it requires a particular test runner. Therefore I wrote a JUnit rule to conditinoally ignore tests. – Michelsen

After installing the junit-ext jar (found here code.google.com/p/junit-ext/downloads/… ) in our local repository and implementing this @RunIf annotation... nothing! It is totally ignored, and I think the reason might be that junit-ext seems to depend on junit 4.5. We need 4.9+ due to spring-test. So... never mind that. – Nightclub

Answer 3

In JUnit 4, another option for you may be to create an annotation to denote that the test needs to meet your custom criteria, then extend the default runner with your own and using reflection, base your decision on the custom criteria. It may look something like this:

public class CustomRunner extends BlockJUnit4ClassRunner {
    public CTRunner(Class<?> klass) throws initializationError {
        super(klass);
    }

    @Override
    protected boolean isIgnored(FrameworkMethod child) {
        if(shouldIgnore()) {
            return true;
        }
        return super.isIgnored(child);
    }

    private boolean shouldIgnore(class) {
        /* some custom criteria */
    }
}

Grovergroves

While this looks nice and clean, it doesn't work with current versions if JUnit4, since the BlockJUnit4ClassRunner doesn't offer the isIgnored method anymore. – Fucoid

Answer 4

Additionally to the answer of @tkruse and @Yishai:
I do this way to conditionally skip test methods especially for Parameterized tests, if a test method should only run for some test data records.

public class MyTest {
    // get current test method
    @Rule public TestName testName = new TestName();
    
    @Before
    public void setUp() {
        org.junit.Assume.assumeTrue(new Function<String, Boolean>() {
          @Override
          public Boolean apply(String testMethod) {
            if (testMethod.startsWith("testMyMethod")) {
              return <some condition>;
            }
            return true;
          }
        }.apply(testName.getMethodName()));
        
        ... continue setup ...
    }
}

Pettus

Answer 5

A quick note: Assume.assumeTrue(condition) ignores rest of the steps but passes the test. To fail the test, use org.junit.Assert.fail() inside the conditional statement. Works same like Assume.assumeTrue() but fails the test.

Nikaniki

As noted in the answers above, a failed assumption does not cause the test to pass, it returns a separate status. Some runners might erroneously report this as if it were a pass, but that's a weakness/bug in the test runner (and the default JUnit runner displays the test as ignored). And as for your final sentence, failing the test is specifically not what I want(ed) to do. – Sherrard

Oh OK. The tests passed on failed assumption in my case but I wanted them to be reported as failed (I was checking for an exception from Test Watcher). Forcing a failure helped me. – Nikaniki