Exceptions thrown while soft asserting fail the subsequent tests

Question

As per title, I'm trying to run a test case in a loop. To be able to calculate the number of failed assertions, I'm expecting that if AssertJ is trying to assert the returned value from a method call, it should softly fail a single iteration and carry on. Otherwise, it defies the purpose of soft assertions. Here's a snippet illustrating this:

    public static void main(String[] args) {
        SoftAssertions softAssertions = new SoftAssertions();
        softAssertions.assertThat(throwException(10)).isTrue();
        softAssertions.assertThat(throwException(10)).isTrue();
        softAssertions.assertThat(throwException(1)).isTrue();
        softAssertions.assertAll();
    }

    private static boolean throwException(int stuff){
        if(stuff == 1){
           throw new RuntimeException();
       }
       return true;
    }

The output:

   Exception in thread "main" java.lang.RuntimeException
    at eLCMUpdate.throwException(MyClass.java:101)
    at eLCMUpdate.main(MyClass.java:95)

I'm missing something here. Am I doing something wrong?

Anschauung

Answer 1

According to my understanding soft assertions work on boolean values and not on exceptions.

Also: if you throw an exception before calling softAssertions.assertAll(), obviously this method will also never be executed. This is actually the cause of the behaviour you reported.

Just try to debug through your code and you will see that the softAssertions.assertAll() is never called.

Soft assertions will work properly if you change your code to:

@Test
void soft_assertions() {
    SoftAssertions softAssertions = new SoftAssertions();
    softAssertions.assertThat(checkCondition(10)).isTrue();
    softAssertions.assertThat(checkCondition(10)).isTrue();
    softAssertions.assertThat(checkCondition(1)).isTrue();
    softAssertions.assertThat(checkCondition(2)).isTrue();
    softAssertions.assertThat(checkCondition(20)).isTrue();
    softAssertions.assertAll();
}

private static boolean checkCondition(int stuff){
    if(stuff == 1 || stuff == 2){
        return false;
    }
    return true;
}

This will output the result of multiple assertions and not stop on the evaluation of the first failed assertion.

Output:

org.assertj.core.api.SoftAssertionError: 
The following 2 assertions failed:
1) 
Expecting:
 <false>
to be equal to:
 <true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:301)
2) 
Expecting:
 <false>
to be equal to:
 <true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:302)

Update

SoftAssertion does not seem to fit your purpose.

I suggest you use instead JUnit 5 assertAll. According to my tests it evaluates all conditions in an assertAll block and survives exceptions too. The problem here is you need JUnit 5 which is probably not largely adopted yet.

Here is an example with a failure on a boolean condition and also an exception. Both are reported in the console.

@Test
void soft_assertions() {
    assertAll("Check condition",
            () -> assertThat(checkCondition(9)).isTrue(),
            () -> assertThat(checkCondition(10)).isTrue(),
            () -> assertThat(checkCondition(11)).isTrue(),
            () -> assertThat(checkCondition(2)).isTrue(), // Throws exception
            () -> assertThat(checkCondition(3)).isFalse(), // fails
            () -> assertThrows(IllegalArgumentException.class, () -> {
                checkCondition(1);
            })
    );
}

private static boolean checkCondition(int stuff) {
    if (stuff == 1 || stuff == 2) {
        throw new IllegalArgumentException();
    }
    return true;
}

You will see this in the output:

org.opentest4j.MultipleFailuresError: Check condition (2 failures)
    <no message> in java.lang.IllegalArgumentException

Expecting:
 <true>
to be equal to:
 <false>
but was not.

Louvre

This was just to test how AssertJ handles exceptions. My actual code has a method call, this method is susceptible to exceptions being thrown. I can always catch the exceptions and handle them, but it seems illogical to me and as I said defies the purpose of soft assertions. – Anschauung

Thanks a lot for the info. For some reason, I don't seem to be able to get the output you got :/ Which assertThat function you're using? It seems to me the one from AssertJ. – Anschauung

Check the JUnit 5 guide examples on assertAll. See junit.org/junit5/docs/current/user-guide . assertAll is what could help you, because it handles exceptions according to what you need. – Louvre

I would love to accept your answer if it is right. I just cannot get the output you got. Can you please help me point what I'm doing wrong? – Anschauung

@enissay, I have create a GitHub repository maven project which contains the code of this test and you can just clone and try out: github.com/gilfernandes/junit5test . Please also not that I have not published the full lengthy output of the test, but I have missed out on some parts of it. But the relevant message is indeed in the post. – Louvre

Answer 2

The problem in the code softAssertions.assertThat(throwException(10)).isTrue(); is that if the exception is thrown then assertThat is not executed at all.

What you need is to lazy evaluate the code you are passing in assertThat, you can do this with AssertJ assertThatCode as below:

final SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThatCode(() -> throwException(10)).doesNotThrowAnyException();
softAssertions.assertThatCode(() -> throwException(1)).isInstanceOf(RuntimeException.class);
softAssertions.assertAll();

Vd

I have seen this option, but the thing is, I cannot predict all the exceptions I might get. Cause if I don't a wild NPE could fail my test. – Anschauung

Nevertheless +1 for the assertThatCode idea. – Louvre