Spock - Testing Exceptions with Data Tables

Question

How can exceptions be tested in a nice way (e.g. data tables) with Spock?

Example: Having a method validateUser that can throw exceptions with different messages or no exception if the user is valid.

The specification class itself:

class User { String userName }

class SomeSpec extends spock.lang.Specification {

    ...tests go here...

    private validateUser(User user) {
        if (!user) throw new Exception ('no user')
        if (!user.userName) throw new Exception ('no userName')
    }
}

Variant 1

This one is working but the real intention is cluttered by all the when / then labels and the repeated calls of validateUser(user).

    def 'validate user - the long way - working but not nice'() {
        when:
        def user = new User(userName: 'tester')
        validateUser(user)

        then:
        noExceptionThrown()

        when:
        user = new User(userName: null)
        validateUser(user)

        then:
        def ex = thrown(Exception)
        ex.message == 'no userName'

        when:
        user = null
        validateUser(user)

        then:
        ex = thrown(Exception)
        ex.message == 'no user'
    }

Variant 2

This one is not working because of this error raised by Spock at compile time:

Exception conditions are only allowed in 'then' blocks

    def 'validate user - data table 1 - not working'() {
        when:
        validateUser(user)

        then:
        check()

        where:
        user                         || check
        new User(userName: 'tester') || { noExceptionThrown() }
        new User(userName: null)     || { Exception ex = thrown(); ex.message == 'no userName' }
        null                         || { Exception ex = thrown(); ex.message == 'no user' }
    }

Variant 3

This one is not working because of this error raised by Spock at compile time:

Exception conditions are only allowed as top-level statements

    def 'validate user - data table 2 - not working'() {
        when:
        validateUser(user)

        then:
        if (expectedException) {
            def ex = thrown(expectedException)
            ex.message == expectedMessage
        } else {
            noExceptionThrown()
        }

        where:
        user                         || expectedException | expectedMessage
        new User(userName: 'tester') || null              | null
        new User(userName: null)     || Exception         | 'no userName'
        null                         || Exception         | 'no user'
    }

Andriette

Came across the same scenario last week and I did exactly what @peter has suggested. :) Handling two vairants of exception (thrown/notThrown) based on one data table is not the way. You cannot even have the thrown exception in the data table. – Gaslit

Answer 1

The recommended solution is to have two methods: one that tests the good cases, and another that tests the bad cases. Then both methods can make use of data tables.

Example:

class SomeSpec extends Specification {

    class User { String userName }

    def 'validate valid user'() {
        when:
        validateUser(user)

        then:
        noExceptionThrown()

        where:
        user << [
                new User(userName: 'tester'),
                new User(userName: 'joe')]
    }

    def 'validate invalid user'() {
        when:
        validateUser(user)

        then:
        def error = thrown(expectedException)
        error.message == expectedMessage

        where:
        user                     || expectedException | expectedMessage
        new User(userName: null) || Exception         | 'no userName'
        new User(userName: '')   || Exception         | 'no userName'
        null                     || Exception         | 'no user'
    }

    private validateUser(User user) {
        if (!user) throw new Exception('no user')
        if (!user.userName) throw new Exception('no userName')
    }

}

Kidd

One question I remember, can't thrown(MyException) return null if MyException is not thrown? – Gaslit

I will have to revisit my test. But I was getting errors using thrown()/notThrown() in the data tables. Thanks anyways for a wonderful test framework. I became a "that-BDD-developer" in my work because of you. ;) – Gaslit

I could potentially work that way, but currently you can't pass null to thrown(). – Kidd

@PeterNiederwieser This would be really useful with a basic example of how to handle exceptions within a data table. This is the top hit for "spock data exceptions" on Google and a reference example (or point to doc) would be very helpful Thanks. – Liebowitz

Answer 2

Here is the solution I came up with. It's basically Variant 3, but it uses a try/catch block to avoid using Spock's exception conditions (since those have to be top level).

def "validate user - data table 3 - working"() {
    expect:
    try {
        validateUser(user)
        assert !expectException
    }
    catch (UserException ex)
    {
        assert expectException
        assert ex.message == expectedMessage
    }

    where:
    user                         || expectException | expectedMessage
    new User(userName: 'tester') || false           | null
    new User(userName: null)     || true            | 'no userName'
    null                         || true            | 'no user'
}

Some caveats:

1. You need multiple catch blocks to test different exceptions.
2. You have to use explicit conditions (assert statements) inside of try/catch blocks.
3. You can't separate your stimulus and responses into when-then blocks.

Doubletongue

Worked perfectly for my situation. I just updated to check exception only if message supplied: assert !exceptionMessage, and can drop the expectException column. – Sane

Answer 3

You can wrap your method call with a method that returns the message or the exception class, or a map of both...

  def 'validate user - data table 2 - not working'() {
        expect:
            expectedMessage == getExceptionMessage(&validateUser,user)
        where:
        user                         || expectedMessage
        new User(userName: 'tester') || null
        new User(userName: null)     || 'no userName'
        null                         || 'no user'
    }

    String getExceptionMessage(Closure c, Object... args){
        try{
            return c.call(args)
            //or return null here if you want to check only for exceptions
        }catch(Exception e){
            return e.message
        }
    }

Doralyn

Answer 4

Here's how I do it, I modify the when: clause to always throw a Success exception, that way you don't need separate tests or logic to tell whether to call thrown or notThrown, just always call thrown with the data table telling whether to expect Success or not.

You could rename Success to be None or NoException or whatever you prefer.

class User { String userName }

class SomeSpec extends spock.lang.Specification {

    class Success extends Exception {}

    def 'validate user - data table 2 - working'() {
        when:
            validateUser(user)
            throw new Success ()

        then:
            def ex = thrown(expectedException)
            ex.message == expectedMessage

        where:
            user                         || expectedException | expectedMessage 
            new User(userName: 'tester') || Success           | null
            new User(userName: null)     || Exception         | 'no userName'
            null                         || Exception         | 'no user'
    }

    private validateUser(User user) {
        if (!user) throw new Exception ('no user')
        if (!user.userName) throw new Exception ('no userName')
    }
}

One extra thing I would change, would be to use a subclass for the failure exceptions too to avoid a Success being accidentally caught when you were really expecting a failure. It doesn't affect your example because you have an extra check for the message, but other tests might just test the exception type.

class Failure extends Exception {}

and use that or some other "real" exception instead of the vanilla Exception

Armil

IMO, throwing an exception indicating success is a pretty bad code smell. See Effective Java Item 69: Use Exceptions only for exceptional conditions. – Boot

It's only in the test that it's thrown, to work around a limitation of the framework, so IMO that type of heuristic doesn't apply, or is outweighed by the other smells that it masks. – Armil

clever! and reads well, to boot – Ulrich

Answer 5

I have solution which not distort your test workflow and you can analyze exception by content of dynamic object placed in where table

@Unroll
def "test example [a=#a, b=#b]"() {
    given:
    def response
    def caughtEx

    when:
    try {
      result = someAmazingFunctionWhichThrowsSometimes(a,b)
    } catch (Exception ex) {
      caughtEx = ex
    }

    then:
    result == expected

    if (exception.expected) {
        assert caughtEx != null && exception.type.isInstance(caughtEx)
    } else {
        assert caughtEx == null
    }

    where:
    a    | b    || exception                                  | expected
    8    | 4    || [expected: false]                          | 2
    6    | 3    || [expected: false]                          | 3
    6    | 2    || [expected: false]                          | 3
    4    | 0    || [expected: true, type: RuntimeException]   | null

}

Matador

Answer 6

Using the example from @AmanuelNega I had a try at this on the spock web console and saved the code at http://meetspock.appspot.com/script/5713144022302720

import spock.lang.Specification

class MathDemo {
    static determineAverage(...values) 
      throws IllegalArgumentException {
        for (item in values) {
            if (! (item instanceof Number)) {
                throw new IllegalArgumentException()
            }
        }

        if (!values) {
            return 0
        }

        return values.sum() / values.size()
    }
}

class AvgSpec extends Specification {

    @Unroll
    def "average of #values gives #result"(values, result){
        expect:
            MathDemo.determineAverage(*values) == result

        where:
            values       || result
            [1,2,3]      || 2
            [2, 7, 4, 4] || 4.25
            []           || 0
    }

    @Unroll
    def "determineAverage called with #values throws #exception"(values, exception){
        setup:
           def e = getException(MathDemo.&determineAverage, *values)

        expect:
            exception == e?.class

        where:
            values       || exception
            ['kitten', 1]|| java.lang.IllegalArgumentException
            [99, true]   || java.lang.IllegalArgumentException
            [1,2,3]      || null
    }

    Exception getException(closure, ...args){
        try{
            closure.call(args)
            return null
        } catch(any) {
            return any
        }
    }
}
​

Peraza

Answer 7

Here is an example of how I achieved it using @Unroll and the when:, then:, and where: blocks. It runs using all 3 of the tests with the data from the data table:

import spock.lang.Specification
import spock.lang.Unroll

import java.util.regex.Pattern

class MyVowelString {
    private static final Pattern HAS_VOWELS = Pattern.compile('[aeiouAEIOU]')
    final String string

    MyVowelString(String string) {
        assert string != null && HAS_VOWELS.matcher(string).find()
        this.string = string
    }
}

class PositiveNumberTest extends Specification {
    @Unroll
    def "invalid constructors with argument #number"() {
        when:
        new MyVowelString(string)

        then:
        thrown(AssertionError)

        where:
        string | _
        ''     | _
        null   | _
        'pppp' | _
    }
}

Emmer