Unit test succeeds in debug mode but fails when running it normally

Question

Why does my unit test succeed in debug mode but fail when running it normally?

public class ExecutorServiceTest extends MockitoTestCase{   
  private int numThreads;
  private ExecutorService pool;
  private volatile boolean interruptedBitSet;

  @Override
  public void setUp() {
    numThreads = 5;
    pool = Executors.newFixedThreadPool(numThreads);
  }

class TaskChecksForInterruptedBit implements Callable<String> {
    @Override
    public String call() throws Exception {
      interruptedBitSet = false;
      while (!Thread.currentThread().isInterrupted()) {
      }
      interruptedBitSet = Thread.currentThread().isInterrupted();
      return "blah";
    }
  }

public void testCancelSetsInterruptedBitInCallable() throws Exception {
    interruptedBitSet = false;
    final Future<String> future = 
        pool.submit(new TaskChecksForInterruptedBit());
    final boolean wasJustCancelled = future.cancel(true);
    assertTrue(wasJustCancelled);

    // Give time for the thread to notice the interrupted bit and set the flag
    Thread.sleep(5000);

    // This succeeds when stepping through w/ a debugger, but fails when running
    // the test straight. WHY?
    assertTrue(interruptedBitSet);

    assertTrue(future.isDone());
    assertTrue(future.isCancelled());
  }
}

Ravens

Suggestion, try making the interruptedBitSet volatile – Dougherty

Where is your breakpoint when you debug? – Constanta

@Constanta I put it on 'interruptedBitSet = false;' – Ravens

Answer 1

You have to make sure your task actually started running. It may get cancelled before it even has a chance.

public class ExecutorServiceTest {
    private int numThreads;
    private ExecutorService pool;
    private volatile boolean interruptedBitSet;
    private static final CountDownLatch latch = new CountDownLatch(1);

    @Before
    public void setUp() {
        numThreads = 5;
        pool = Executors.newFixedThreadPool(numThreads);
    }

    class TaskChecksForInterruptedBit implements Callable<String> {
        @Override
        public String call() throws Exception {
            interruptedBitSet = false;
            latch.countDown();
            while (!Thread.currentThread().isInterrupted()) {
                System.out.println(System.currentTimeMillis());
            }
            System.out.println("haha");
            interruptedBitSet = Thread.currentThread().isInterrupted();
            return "blah";
        }
    }

    @Test
    public void testCancelSetsInterruptedBitInCallable() throws Exception {
        final Future<String> future =
                pool.submit(new TaskChecksForInterruptedBit());
        interruptedBitSet = false;
        latch.await();
        final boolean wasJustCancelled = future.cancel(true);
        Assert.assertTrue(wasJustCancelled);

        // Give time for the thread to notice the interrupted bit and set the flag
        Thread.sleep(5000);

        // This succeeds when stepping through w/ a debugger, but fails when running
        // the test straight. WHY?
        Assert.assertTrue(interruptedBitSet);

        Assert.assertTrue(future.isDone());
        Assert.assertTrue(future.isCancelled());
    }
}

Phyto

Answer 2

The reason is almost certainly that your breakpoint in the debugger is halting the main thread but not any of the background threads - the ones in the ExecutorService. When debugging in eclipse you can change the breakpoint to halt all threads instead of just the main one.

When not debugging the submission of the task and the immediate cancellation are so quick that you are cancelling the task before it even runs once. Try adding a sleep delay between these lines:

final Future<String> future =  pool.submit(new TaskChecksForInterruptedBit());
Thread.sleep(1000);
final boolean wasJustCancelled = future.cancel(true);

Constanta

Why doesn't the main thread see the modified flag (modified by the spawned thread) in release mode? – Ravens

Let me clarify: I put it on 'interruptedBitSet = false' in the test method, not the Callable's call() method. – Ravens

@Chris-Morris sorry, I misread the code first, I've edited my answer with another suggestion – Constanta

Answer 3

I know this is old but I just had the same problem. My issue was that I had an IEnumerable that I was enumerating and checking the output.

When running the Unit test, the IEnumerable was returning a different ordering that when debugging. This is the nature of IEnumerable and simply adding an OrderBy clause solved my problem.

I hope this helps someone out there as it can be a frustrating problem to find.

Sleuth

Gotcha, reflections.getSubTypesOf(Foo.class) orders differently in Run / Debug – Extramarital

Answer 4

You should check whether all the threads dies before terminating the main thread

private void shutdownExecutionService(ExecutorService executorService) {
    if (executorService != null) {
        try {
            executorService.shutdown();
            while (!executorService.awaitTermination(10, TimeUnit.HOURS)) {
                logger.info("Awaiting completion of threads.");
            }
        } catch (final InterruptedException e) {
            logger.error("Error while shutting down threadpool", e);
        }
    }
}

Bordelon

Answer 5

I was having a similar issue while running a homework assignment from an online course. The grader program from the course which I added to the build path used JUnit4, my version of Eclipse added JUnit5 to any new test cases. I've created a new Java Project and added JUnit5 to the build bath for my test cases without the grader and it fixed it for me. Hope this helps.

Anselmi