Ordering unit tests in Eclipse's JUnit view

Question

The JUnit view in Eclipse seems to order the tests randomly. How can I order them by class name?

Etan

This drives me nuts in Eclipse, and I wish someone would add this as an option to the plugin. I wish the accepted answer actually fixed the problem! – Gereron

Answer 1

As Gary said in the comments:

it would be nice if Unit Runner could be told to go ahead and order them by class name. Hmm, maybe I should look into the source code...

I did look but there's no hint of a functionality to sort these names. I would suggest a change request to the JUnit plugin, but I don't think, that there are lot of people using this thing, so: DIY.

I would like to see the solution if you modify the plugin code.

Spinozism

Yeah, probably the best solution is to modify the plugin code. Wish I had the time. – Etan

Answer 2

One thing that one might do is using the schema of JUnit 3.x. We used a test suite that was called AllTests where you add the tests to it in a specific order. And for every package we got another AllTests. Giving those test suites a name being the same as the package enables one to easily build a hierarchy that should be valued by the junit plugin.

I really dislike how it is even presenting the test methods inside the Junit viewer. It should be in the very same order as they are specified in the TestCase class. I order those methods in the way of importance and features. So the upmost failing method is to correct first and then the more special one in the later part of the test case.

That is really annoying that the test runner is scrambling those. I will take a look at it myself and if I find a solution I will update this answer.

Update:

My problem with the ordering of method names within a TestCase is related to this one: https://bugs.java.com/bugdatabase/view_bug?bug_id=7023180 (Thanks Oracle!).

So in the end oracle changed the ordering of the methods within a class.getMethods or class.getDeclaredMethods call. Now the methods are random and can change between different runs of the JVM. It seams to be related to optimizations of compare or even is an attempt to compress method name - who knows... .

So whats left. First one can use: @FixMethodOrder (from javacodegeeks.com):

1. @FixMethodOrder(MethodSorters.DEFAULT) – deterministic order based on an internal comparator
2. @FixMethodOrder(MethodSorters.NAME_ASCENDING) – ascending order of method names
3. @FixMethodOrder(MethodSorters.JVM) – pre 4.11 way of depending on reflection based order

Well that is stupid but explains why people start using test1TestName schema.

Update2:

I use ASM since Javassist also produces random sorted methods on getMethods(). They use Maps internally. With ASM I just use a Visitor.

package org.junit.runners.model;

import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;

import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassVisitor;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;

import com.flirtbox.ioc.OrderTest;

/**
 * @author Martin Kersten
*/
public class TestClassUtil {
public static class MyClassVisitor extends ClassVisitor {
    private final List<String> names;
    public MyClassVisitor(List<String> names) {
        super(Opcodes.ASM4);
        this.names = names;
    }
    
    @Override
    public MethodVisitor visitMethod(int access, String name, String desc,
            String signature, String[] exceptions) {
        names.add(name);
        return super.visitMethod(access, name, desc, signature, exceptions);
    }
}

private static List<String> getMethodNamesInCorrectOrder(Class<?> clazz) throws IOException {
    InputStream in = OrderTest.class.getResourceAsStream("/" + clazz.getName().replace('.', '/') + ".class");
    ClassReader classReader=new ClassReader(in);
    List<String> methodNames = new ArrayList<>();
    classReader.accept(new MyClassVisitor(methodNames), 0);
    return methodNames;
}

public static void sort(Class<?> fClass, List<FrameworkMethod> list) {
    try {
        final List<String> names = getMethodNamesInCorrectOrder(fClass);
        Collections.sort(list, new Comparator<FrameworkMethod>() {
            @Override
            public int compare(FrameworkMethod methodA, FrameworkMethod methodB) {
                int indexA = names.indexOf(methodA.getName());
                int indexB = names.indexOf(methodB.getName());
                if(indexA == -1)
                    indexA = names.size();
                if(indexB == -1)
                    indexB = names.size();
                return indexA - indexB;
            }
        });
    } catch (IOException e) {
        throw new RuntimeException("Could not optain the method names of " + fClass.getName() + " in correct order", e);
    }
}
}

Just put this in your src/test/java folder in the package org.junit.runners.model. Now copy the org.junit.runners.model.TestClass of the junit 4.5 lib to the same package and alter its constructor by adding the sorting routine.

 public TestClass(Class<?> klass) {
    fClass= klass;
    if (klass != null && klass.getConstructors().length > 1)
        throw new IllegalArgumentException(
                "Test class can only have one constructor");

    for (Class<?> eachClass : getSuperClasses(fClass))
        for (Method eachMethod : eachClass.getDeclaredMethods())
            addToAnnotationLists(new FrameworkMethod(eachMethod));
    
            //New Part
    for(List<FrameworkMethod> list : fMethodsForAnnotations.values()) {
        TestClassUtil.sort(fClass, list);
    }
    
    //Remove once you have verified the class is really picked up
    System.out.println("New TestClass for " + klass.getName());

}

Here you go. Now you have nicely sorted methods in the order they are declared within the java file. If you wonder the class path is usually set that way that everything in your src (target or bin) folder is considered first by the classloader. So while defining the very same package and the same class you can 'override' every class / interface in any library you use. Thats the trick!

Update3 I was able to get a tree view of every package and every class in the right order to.

• The idea is to subclass ParentRunner and then add all classes to it that you identify as being public and having methods annotated with test.
• Add a getName() method returning only the package name of the class your suite runner is representing (so you see the tree as a package tree without the suite's class name).
• Inspect subdirectories if you find a certain suite class (I use AllTests for all suite classes).
• If you do not find a suite class in a subdirectory check all of its subdirectories, this way you dont miss a package containing tests if the parent directory is not containing a suite.

That was it. The suite class I add everywhere is: @RunWith(MySuiteRunner.class) public class AllTests { }

That's it. It should you give enough to start and extend on this one. The suite runner is only using reflection but I sort the test classes and suits of the subdirectories alphabetically and suits of subdirectories (which represent the packages they are in) are sorted upmost.

Elm

Answer 3

11 years later, the JUnit view does have name ordering, and just got "Execution time" ordering as well.
See Eclipse 4.17 (2020-09)

Sort test results by execution time

JUnit view now provides the ability to sort results by execution time.

By default, results will be sorted by execution order.
Choosing Sort By > Execution Time from the JUnit View menu will reorder the results once all tests are complete.

While tests are still running, they will be shown in execution order.

Sorting by execution order results in:

Danell

Answer 4

mark wrote:

it orders them base on execution time, maybe you should sort your methods? source/sort members

mark is right. But you cannot sort your unit test. It's not allowed to speculate about the order of execution.

Unit tests have to be built independently and it's random, how they are called by the UnitRunner.

In most cases, the test methods are sorted alphabetically. The classes are random. Try to use a TestSuite to order your tests.

Spinozism

Well, I'm trying to avoid using TestSuite. I appreciate unit test independence, and I maintain that when creating them, but it would be nice if Unit Runner could be told to go ahead and order them by class name. Hmm, maybe I should look into the source code... – Etan

Answer 5

If you really need hard dependency between your JUnit test, try JExample extension

JExample introduces producer-consumer relationships to unit-testing.
A producer is a test method that yields its unit under test as return value.
A consumer is a test method that depends on one or more producers and their return values.

You can install it in Eclipse, for Junit4.4 or 4.5.

import jexample.Depends;

  @Test
  @Depends("#testEmpty") 
  public Stack<Integer> testPush(Stack<Integer> $) { 
      $.push(42); 
      assertFalse($.isEmpty()); 
      return $; 
  } 

---

As mentioned in this IBM article "In pursuit of code quality: JUnit 4 vs. TestNG":

One thing the JUnit framework tries to achieve is test isolation.
On the downside, this makes it very difficult to specify an order for test-case execution, which is essential to any kind of dependent testing.
Developers have used different techniques to get around this, like specifying test cases in alphabetical order or relying heavily on fixtures (@Before @After) to properly set things up.

These workarounds are fine for tests that succeed, but for tests that fail, they have an inconvenient consequence: every subsequent dependent test also fails. In some situations, this can lead to large test suites reporting unnecessary failures

So beware: if you retain any solution for ordering your JUnit tests the way you want... you need to think if that solution support a "skip" feature in order to allow other tests to proceed even if one of them fails.

Danell

I think he just want to see the results in a specific order. – Spinozism

... yes hence my answer: since you can not sort the results, you can try to order the tests. – Danell

Yeah, just the results. And only because I might want to look for a specific test result after the unit tests have ran. I'm not sure why Eclipse can't order the test results as they're being run. Just use a SortedSet. – Etan

Answer 6

Ordering tests in JUnit view has been filed as bug #386453 in Eclipse Bugzilla. Commenting and/or voting there may help to get more visibility to this problem.

Cimmerian

Answer 7

I was also searching for a solution for this, and I found a kind of crack from the below URL. I don't know whether it works for you or not, but it worked for me in Spring Tool Suite 2.5.2.

http://osdir.com/ml/java.junit.user/2002-10/msg00077.html

Mixon