JUnit's @TestMethodOrder annotation not working

Question

I'm having a problem with following integration test

import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;

@SpringBootTest
@ActiveProfiles("test")
@TestMethodOrder(OrderAnnotation.class)
public class FooServiceIT {
    @Test
    @Order(1)
    void testUploadSuccess() { ... }
    @Test
    @Order(2)
    void testDownloadSuccess() { ... }
    @Test
    @Order(3)
    void testDeleteSuccess() { ... }
}

I'd expect when I run the test that the execution order would 1, 2, 3 but for some reason, the actual execution order is 2, 3, 1.

Tbh, I'm clueless why the annotation is not working. I'm using Spring Boot 2.1.3 with JUnit 5.4.

Sabu

theoretically you should write your tests the way that they can run in any order... why do you need to order them? – Spirituality

You have configured wrongly. Try to put @TestMethodOrder(MethodOrderer.OrderAnnotation.class) https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/TestMethodOrder.html – Tweezers

@Tweezers I've updated the imports, I was already using the OrderAnnotation class properly. @RuthiRuth the methods are manipulating files on a remote server, obviously, there the download/delete method will fail if I don't upload the file on the server first (I'd also like to keep the method separate - i.e. not calling the upload method in the download test). – Sabu

Can you please share the rest of your imports, in particular for the @Test annotation? – Bowl

One reason for ordering tests is to resolve cases where running single tests in a suite succeeds but running multiple tests fails on, say the second one because the test harness has not been torn down after the first test as expected. This can be a real pain to debug, especially with the latest versions of the Maven test plugins, where the fork configurations seem sometimes to interact non deterministically with junit 5. – Altruism

Use this annotation @TestMethodOrder(MethodOrderer.OrderAnnotation.class) – Mallette

Answer 1

You need to configure correctly your IDE.

Requirements

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0</version>
</dependency>

Do not use JUnit 5 that offers your IDE. If you add it as library, you will get:

No tests found for with test runner 'JUnit 5' 
==================== and this exception ===================
TestEngine with ID 'junit-vintage' failed to discover tests
java.lang.SecurityException: class "org.junit.jupiter.api.TestMethodOrder"'s signer information does not match signer information of other classes in the same package

So just include mentioned dependency only and your code will work as you expect:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class FooServiceIT {

    @Test
    @Order(1)
    public void testUploadSuccess() {
        System.out.println("1");
    }

    @Test
    @Order(2)
    public void testDownloadSuccess() {
        System.out.println("2");
    }

    @Test
    @Order(3)
    public void testDeleteSuccess() {
        System.out.println("3");
    }
}

JUnit result:

1
2
3

Tweezers

Thanks! This wasn't the actual problem but your answer pointed me to right direction. The problem was that I've used the junit-jupiter artifactId as an aggregator dependency which for some reason did not contain the junit-jupiter-engine in the 5.4.0 version but in 5.3.2 version. After I've used the dependencies in correct versions, the tests did not run at all but adding platform launcher as suggested in https://mcmap.net/q/452717/-nosuchmethoderror-org-junit-platform-commons-util-reflectionutils-trytoloadclass solved the issue. Now the tests works, and they work in correct order. – Sabu

You just saved me from being extra tilted due to a wrong dependency version. Cheers! – Perhaps

I am having the same issue. Just replaced junit-jupiter-engine version to 5.4.0 and still having the reverse order execution. Can you please provide some feedback? Many thanks. – Nobles

Just want to add, this thread github.com/junit-team/junit5/issues/1776 also mentioned the same thing. Still No luck - any help would be appreciated. Thanks. – Nobles

@Nobles make sure junit-jupiter-engine is taking correct version, not from IDE. If you can share your code, I can take a look – Tweezers

@Tweezers - I've just confirmed that using the junit-jupiter-engine correct version. Please see yourself. com.personalitytest.demo:personalitytest:jar:1.0-SNAPSHOT \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0:test – Nobles

@Tweezers - please check the code which I've posted in the separate thread. Thanks. – Nobles

junit-jupiter-engine comes with the spring-boot-starter-test dependency. no need to add it extra. please check if you annotated the class with @TestMethodOrder(MethodOrderer.OrderAnnotation.class) before adding another dependency – Mehitable

Using a recent Eclipse (2021-09 (4.21.0)), the built-in junit-jupiter-engine (appears to be 5.7.1) is good enough, all that is required is the @TestMethodOrder annotation. – Brumbaugh

Answer 2

I have faced the same issue. But, I found the problem where exactly is on my case. Wrongly import the "Order" class.

Wrong One

import org.springframework.core.annotation.Order;

Right one

*import org.junit.jupiter.api.Order;*

Also, please verify the following five classes with proper import

import org.junit.jupiter.api.Assertions;

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;

import org.junit.jupiter.api.Order;

import org.junit.jupiter.api.Test;

import org.junit.jupiter.api.TestMethodOrder;

Cecilececiley

dude...you saved me lots of hours man. thank you for this. – Duax

Answer 3

From the docs https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/TestMethodOrder.html

 @TestMethodOrder(MethodOrderer.OrderAnnotation.class)
 class OrderedTests {

     @Test
     @Order(1)
     void nullValues() {}

     @Test
     @Order(2)
     void emptyValues() {}

     @Test
     @Order(3)
     void validValues() {}
 }

Poltroonery

How your answer addresses the problem? – Amethyst

This is the thing which I was missing. Thank you! – Attested

Answer 4

If you have/had JUnit 4, check the import for annotation @Test: import org.junit.Test;

For JUnit 5 import should be: import org.junit.jupiter.api.Test;

It was my issue for this question

Infarction

Answer 5

Please make sure you import libraries from org.junit.jupiter.api instead of org.junit

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(OrderAnnotation.class)
class OrderedTestsDemo {

    @Test
    @Order(1)
    void nullValues() {
        // perform assertions against null values
    }

    @Test
    @Order(2)
    void emptyValues() {
        // perform assertions against empty values
    }

    @Test
    @Order(3)
    void validValues() {
        // perform assertions against valid values
    }

}

Code Sources: https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order-methods

Maeve

Answer 6

you need to configure correctly your pom.xml. See mine :

<properties>
    <!-- Dependency versions -->
    <junit.jupiter.version>5.6.0</junit.jupiter.version>
    <maven-surefire-plugin.version>3.0.0-M4</maven-surefire-plugin.version>

    <!-- Java 10 -->
    <maven.compiler.source>1.10</maven.compiler.source>
    <maven.compiler.target>1.10</maven.compiler.target>

    <!-- Encoding -->
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

<!-- Jupiter API for writing tests -->
<dependencies>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-engine</artifactId>
        <version>${junit.jupiter.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<!-- Maven Surefire plugin to run tests -->
<build>
    <plugins>
        <!-- plugin to run test cases from maven -->
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin.version}</version>
        </plugin>
        <!-- Maven plugin to use perticular java version to compile code -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.1</version>
            <configuration>
                <source>${maven.compiler.source}</source>
                <target>${maven.compiler.target}</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Now your Junit 5 annotations must function correctly

Florance

This solution just tried and doesn't work – Amu

Answer 7

After applying all the settings based on previous recommended solution. I'm still having the same reverse or random @order execution.

Pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.3.RELEASE</version>
</parent>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0</version>
    <scope>test</scope>
</dependency>

Maven Dependency Tree:

mvn dependency:tree -Dverbose -Dincludes=org.junit.jupiter:junit-jupiter-engine

[INFO] com.personalitytest.demo:personalitytest:jar:1.0-SNAPSHOT
[INFO] \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0:test

JUnit Test:

@SpringBootTest
@ActiveProfiles("test")
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class JUnitOrderTest {

    private static final Logger log = LoggerFactory.getLogger(JUnitOrderTest.class);

    @Test
    @Order(1)
    public void testUploadSuccess() {
        log.info("Junit - Order 1");
    }

    @Test
    @Order(2)
    public void testDownloadSuccess() {
        log.info("Junit - Order 2");
    }

    @Test
    @Order(3)
    public void testDeleteSuccess() {
        log.info("Junit - Order 3");
    }
}

Output:

Running com.personalitytest.demo.security.JUnitOrderTest
08:48:35.470 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 2
08:48:35.480 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 3
08:48:35.481 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 1

Nobles

@Tweezers - Please have a look to the code. Any help would be appreciated - Thanks. – Nobles

@Tweezers - It was strange for me, before I noticed that you're running the tests against Junit5. @MethodExecutionOrder only work with Junit5 and any version of SpringBoot below 2.0 uses Junit4 for testing purposes. So I excluded the Junit4 from Spring settings and installed the Jupiter (Junit5) along MockitoJupiter(Mockito5) and now everything works as per expectation. Thanks for your help much appreciated! – Nobles

Just want to add for other's, @Test annotation works differently for both Junit4 and Junit5. When you've spring boot configured below than 2.0, then it uses Junit4 and you don't realise it while running the tests through mvn clean test. – Nobles

Answer 8

same issue.

For my case. First, it's the @Test annotation, should be org.junit.jupiter.api.Test, not org.junit.Test.

Second, I added @RunWith(SpringRunner.class) in the head of the test class, after I remove it, the order of test cases run as expected.

Standford

RunWith comes with JUnit4, I am not sure it is a good idea to mix up JUnit 4 & 5 features. Might be better to use @ExtendWith(SpringExtension.class) – Benzvi

Answer 9

For Spring Boot 3 users of spring-boot-starter-test and spring-boot-starter-parent POM 3.2.1:

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)

gave me the error

"Class<OrderAnnotation> cannot be resolved to a type" 

even with the import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;

I had to use the full class name in the annotation:

import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation; // USELESS as full name bellow
import org.springframework.boot.test.context.SpringBootTest;


@SpringBootTest
@TestMethodOrder(org.junit.jupiter.api.MethodOrderer.OrderAnnotation.class)
class JUnitOrderTest {
    
    private static final Logger log = LoggerFactory.getLogger(JUnitOrderTest.class);


    @Test
    @Order(1)    
    void zzz() {
        log.info("Junit - Order 1");
    }
    
    @Test
    @Order(2)    
    void aaa() {
        log.info("Junit - Order 2");
    }
    


}

Sixpack

Answer 10

For example, by using Spring Boot you can use @FixMethodOrder(MethodSorters.JVM) instead of @TestMethodOrder(MethodOrderer.OrderAnnotation.class). All of tests are run in the order they appear.

@FixMethodOrder(MethodSorters.JVM)
public class MyTest{

   @Test
   public void zzzz(){}

   @Test
   public void cccc(){}

   @Test
   public void aaaa(){}

   @Test
   public void bbbb(){}
}

Order execution is:

zzzz()
cccc()
aaaa()
bbbb()

Chaldea

Not true. Read the documentation. "Note that the order from the JVM may vary from run to run" – Ymir

@FixMethodOrder is in Junit4 not Junit5 ! – Pasquale