Missing branches when using assertTrue instead of assertNull

Question

In Java/Junit, I need to test for null with some object. There are a variety of ways I can test a condition but I have been using assertTrue for most of my tests. When I check for nulls in an assertTrue, EclEmma states that it is only testing one branch.

When I resolve the statement into a variable manually (like setting the result to a boolean and passing it into assertTrue) the code coverage is deemed complete on the assert but not on the variable initializing line.

Why is this happening? Is this related to the extra byte code that Java apparently adds as mentioned on http://sourceforge.net/apps/trac/eclemma/wiki/FilteringOptions? Any solutions (besides using other assert statements).

assertTrue:

assertTrue( myObject == null ); //1 of 2 branches

assertTrue:

boolean test = (myObject == null); //1 of 2 branches missing
assertTrue(test); // complete

assertNull:

assertNull( myObject ) //complete;

Ware

What is wrong with asserting null by using assertNull? It is usually better to use the appropriate assert type for the thing that is asserted. You better see what's wrong when the test fails without digging too much into the test code. – Unity

@Unity This is true and I have no problem using assertNull if it is a requirement, however, IMO, all assert "types" are just another form of assertTrue. There may be added benefits to using different assert types like added error info/readability but that isn't really an issue here. – Ware

@Unity That isn't to say that I refuse to use assertNull and probably will be using it, I just found this particular problem interesting as I'm basically testing for the same thing but Emma gives completely different results. – Ware

Why are you running code coverage on your test code? I don't think anyone should care if all your test code branches are tested; that seems silly. – Coffeehouse

The underlying issue is unrelated to running coverage on your test code, and unrelated to the use of assertNull versus assertTrue. – Mattson

Answer 1

For most Boolean expressions, the Java compiler generates extra branches in the byte code. JaCoCo produces "branch coverage" based on the generated byte code, not based on the original Java code, and hence shows additional branch coverage information for almost any Boolean expression you would use.

In your code, the Boolean expression you use is myObject == null.

To compute this value, the Java compiler generates code pushing the two arguments on the stack, and then doing a conditional jump in order to push 1 (true) or 0 (false) on the stack. JaCoCo reports the branch coverage of this conditional jump.

Thus, the fact that you use myObject == null triggers the behavior you describe.

As some other examples, try this:

boolean t = true;
boolean f = false;
boolean result1 = (t && f) || f; // 3 out of 6 missed.
boolean result2 = !t;            // 1 out of 2 missed.

This can be useful if the Boolean expression is, for example, returned by a function, which is used as condition in an if-then-else statement somewhere else. While mostly a consequence of the way the Java compiler works, it helps to assess condition coverage (instead of mere branch coverage) of the original Java code.

This feature isn't too well documented, but here are some pointers:

• The JaCoCo test cases for Boolean Expressions
• A JaCoCo forum discussion on branches generated for the statement a = !a
• does anyone have a pointer to more documentation?

So it is, indeed, related to the extra byte code is generated, but not to the specific examples of synthetic byte compiler constructs for which the filtering options are intended.

NOTE: Did major EDIT since initial answer was too much of a guess. Thanks to @ira-baxter for good & critical discussion.

Mattson

Answer 2

The fact that Emma treats a conditional expression as "something with a branch" for (branch) coverage counting IMHO seems simply broken. It isn't a conditional branch.

We can argue more about the Assert; if it were defined as "throws exception on assert failure" then it really does have a conditonal branch; if it is defined [as I think i does, I'm not a Java expert] as "terminate my program on assert failure" then it isn't really a branch. Also obscure are method calls; these are conditional branches in the sense that if the called method throws an exception, control flow does not continue to the "rest of the statement".

Our Java Test Coverage tool gets the (branch) coverage analysis on such conditionals "right".

Variola

There is nothing special about the assertTrue / assertNull methods from the JUnit API. They just throw a java.lang.AssertionError if the condition doesn't hold. This is not directly related to Java's assert keyword. – Mattson

I agree that calling this Emma/JaCoCo feature branch coverage is (very) confusing. (Separate) coverage info telling me for any Boolean (sub)expression whether it evaluated to both true and false is a useful feature though. – Mattson

@avandeursen: determinining whether every condition in a program has been exercised as "true" or "false" isn't branch coverage; it is "condition coverage". Certainly our tool does not do condition coverage; based on past understanding, I do not believe that Emma does condition coverage, but I could be suprised. – Variola

Thanks Ira, you made me think, and I did a major edit of the answer. – Mattson

Answer 3

To get 100% code coverage on boolean methods, do the following

Class RecordService{


    public boolean doesRecordExist(String id){

    return id!=null;

    }


    }

    //Method inside your mock
    @Test
    public boolean testDoesRecordExist(){
    RecordService recordService = mock(RecordService.class);
    when(recordService.doesRecordExists()).thenReturn(
                    anyString()).thenReturn(null);

    }

Bughouse