Handling Exceptions in Python Behave Testing framework

Question

I've been thinking about switching from nose to behave for testing (mocha/chai etc have spoiled me). So far so good, but I can't seem to figure out any way of testing for exceptions besides:

@then("It throws a KeyError exception")
def step_impl(context):
try:
    konfigure.load_env_mapping("baz", context.configs)
except KeyError, e:
    assert (e.message == "No baz configuration found") 

With nose I can annotate a test with

@raises(KeyError)

I can't find anything like this in behave (not in the source, not in the examples, not here). It sure would be grand to be able to specify exceptions that might be thrown in the scenario outlines.

Anyone been down this path?

Pt

It seem to me that ensuring that code throws certain exceptions under certain scenarios is a pretty standard thing to test for. Also good for showing client code the behaviors it can expect. When I test for that, I don't the tests to fail! In any case it's a pretty standard feature of most testing frameworks. – Pt

Maybe, you could help out with #67338970 . Or at least with my question about util.show_gherkin_error(msg) - see my comment to Mary Bergman's answer. – Sicyon

Answer 1

This try/catch approach by Barry works, but I see some issues:

• Adding a try/except to your steps means that errors will be hidden.
• Adding an extra decorator is inelegant. I would like my decorator to be a modified @where

My suggestion is to

• have the expect exception before the failing statement
• in the try/catch, raise if the error was not expected
• in the after_scenario, raise error if expected error not found.
• use the modified given/when/then everywhere

Code:

    def given(regexp):
        return _wrapped_step(behave.given, regexp)  #pylint: disable=no-member

    def then(regexp):
        return _wrapped_step(behave.then, regexp)  #pylint: disable=no-member

    def when(regexp):
        return _wrapped_step(behave.when, regexp) #pylint: disable=no-member


    def _wrapped_step(step_function, regexp):
        def wrapper(func):
            """
            This corresponds to, for step_function=given

            @given(regexp)
            @accept_expected_exception
            def a_given_step_function(context, ...
            """
            return step_function(regexp)(_accept_expected_exception(func))
        return wrapper


    def _accept_expected_exception(func):
        """
        If an error is expected, check if it matches the error.
        Otherwise raise it again.
        """
        def wrapper(context, *args, **kwargs):
            try:
                func(context, *args, **kwargs)
            except Exception, e:  #pylint: disable=W0703
                expected_fail = context.expected_fail
                # Reset expected fail, only try matching once.
                context.expected_fail = None
                if expected_fail:
                    expected_fail.assert_exception(e)
                else:
                    raise
        return wrapper


    class ErrorExpected(object):
        def __init__(self, message):
            self.message = message

        def get_message_from_exception(self, exception):
            return str(exception)

        def assert_exception(self, exception):
            actual_msg = self.get_message_from_exception(exception)
            assert self.message == actual_msg, self.failmessage(exception)
        def failmessage(self, exception):
            msg = "Not getting expected error: {0}\nInstead got{1}"
            msg = msg.format(self.message, self.get_message_from_exception(exception))
            return msg


    @given('the next step shall fail with')
    def expect_fail(context):
        if context.expected_fail:
            msg = 'Already expecting failure:\n  {0}'.format(context.expected_fail.message)
            context.expected_fail = None
            util.show_gherkin_error(msg)
        context.expected_fail = ErrorExpected(context.text)

I import my modified given/then/when instead of behave, and add to my environment.py initiating context.expected fail before scenario and checking it after:

    def after_scenario(context, scenario):
        if context.expected_fail:
            msg = "Expected failure not found: %s" % (context.expected_fail.message)
            util.show_gherkin_error(msg)

Ssm

What is util.show_gherkin_error(msg) in context of Python 3 and Behave 1.2.6? – Sicyon

Answer 2

I'm pretty new to BDD myself, but generally, the idea would be that the tests document what behaves the client can expect - not the step implementations. So I'd expect the canonical way to test this would be something like:

When I try to load config baz
Then it throws a KeyError with message "No baz configuration found"

With steps defined like:

@when('...')
def step(context):
    try:
        # do some loading here
        context.exc = None
    except Exception, e:
        context.exc = e

@then('it throws a {type} with message "{msg}"')
def step(context, type, msg):
    assert isinstance(context.exc, eval(type)), "Invalid exception - expected " + type
    assert context.exc.message == msg, "Invalid message - expected " + msg

If that's a common pattern, you could just write your own decorator:

def catch_all(func):
    def wrapper(context, *args, **kwargs):
        try:
            func(context, *args, **kwargs)
            context.exc = None
        except Exception, e:
            context.exc = e

    return wrapper

@when('... ...')
@catch_all
def step(context):
    # do some loading here - same as before

Ornithosis

Wound up abandoning behave. Having the separate cucumber spec file and the convention that forces almost everything to be a multi-step test is just too much work. I found it wrecking my development flow. Back to nose! – Pt

Answer 3

The try / except approach you show is actually completely correct because it shows the way that you would actually use the code in real life. However, there's a reason that you don't completely like it. It leads to ugly problems with things like the following:

Scenario: correct password accepted
Given that I have a correct password
When I attempt to log in  
Then I should get a prompt

Scenario: incorrect password rejected
Given that I have an incorrect password
When I attempt to log in 
Then I should get an exception

If I write the step definition without try/except then the second scenario will fail. If I write it with try/except then the first scenario risks hiding an exception, especially if the exception happens after the prompt has already been printed.

Instead those scenarios should, IMHO, be written as something like

Scenario: correct password accepted
Given that I have a correct password
When I log in  
Then I should get a prompt

Scenario: correct password accepted
Given that I have a correct password
When I try to log in 
Then I should get an exception

The "I log in" step should not use try; The "I try to log in" matches neatly to try and gives away the fact that there might not be success.

Then there comes the question about code reuse between the two almost, but not quite identical steps. Probably we don't want to have two functions which both login. Apart from simply having a common other function you call, you could also do something like this near the end of your step file.

@when(u'{who} try to {what}')
def step_impl(context):
    try:
        context.execute_steps("when" + who + " " + what)
        context.exception=None
    except Exception as e:
        context.exception=e

This will automatically convert all steps containing the word "try to" into steps with the same name but with try to deleted and then protect them with a try/except.

There are some questions about when you actually should deal with exceptions in BDD since they aren't user visible. It's not part of the answer to this question though so I've put them in a separate posting.

Pectinate

Answer 4

Behave is not in the assertion matcher business. Therefore, it does not provide a solution for this. There are already enough Python packages that solve this problem.

SEE ALSO: behave.example: Select an assertion matcher library

Residuum

Hi @Residuum ; thanks for behave BTW I think that the problem here isn't just matchers. It isn't clear when you run your "When" step what you should match on. e.g. if you match on an exception but don't test for it it's possible to mask a different exception later. If you get an exception in a "when" step then it should propagate as a test error. – Pectinate