How do I get SpecFlow to expect an exception?

Question

I'm using SpecFlow, and I'd like to write a scenario such as the following:

Scenario: Pressing add with an empty stack throws an exception
    Given I have entered nothing into the calculator
    When I press add
    Then it should throw an exception

It's calculator.Add() that's going to throw an exception, so how do I handle this in the method marked [Then]?

Phyllis

Hey did you find any of these answers useful? – Kasey

@scoarescoare: Yeah. The problem is that the correct answer, containing all of the required information, is a combination of yours and Kjetil's. Your answer says that my language is wrong, and Kjetil's actually says how to get the exception (or other output) from When to Then. – Phyllis

Thanks for asking this. I have found myself wondering the same thing! – Catling

Answer 1

Great question. I am neither a bdd or specflow expert, however, my first bit of advice would be to take a step back and assess your scenario.

Do you really want to use the terms "throw" and "exception" in this spec? Keep in mind the idea with bdd is to use a ubiquitous language with the business. Ideally, they should be able to read these scenarios and interpret them.

Consider changing your "then" phrase to include something like this:

Scenario: Pressing add with an empty stack displays an error
    Given I have entered nothing into the calculator
    When I press add
    Then the user is presented with an error message

The exception is still thrown in the background but the end result is a simple error message.

Scott Bellware touches this concept in this Herding Code podcast: http://herdingcode.com/?p=176

Kasey

I would add that BDD tools such as specflow are mean to be used in conjunction with TDD. So you write your spec like this and then you would write a unit test that expects an exception. – Percutaneous

Great answer! I'm no expert either but it seems the appropriate response. – Catling

Answer 2

As a newbie to SpecFlow I won't tell you that this is the way to do it, but one way to do it would be to use the ScenarioContext for storing the exception thrown in the When;

try
{
    calculator.Add(1,1);
}
catch (Exception e)
{
    ScenarioContext.Current.Add("Exception_CalculatorAdd", e);
}

In your Then you could check the thrown exception and do asserts on it;

var exception = ScenarioContext.Current["Exception_CalculatorAdd"];
Assert.That(exception, Is.Not.Null);

With that said; I agree with scoarescoare when he says that you should formulate the scenario in a bit more 'business-friendly' wordings. However, using SpecFlow to drive the implementation of your domain-model, catching exceptions and doing asserts on them can come in handy.

Btw: Check out Rob Conery's screencast over at TekPub for some really good tips on using SpecFlow: http://tekpub.com/view/concepts/5

Hasid

In specflow 1.7.1.0, you can reference ScenarioContext.Current.TestError for exceptions caught during a scenario. – Kasey

In my context I have two types of Whens : The normal that throws exceptions like When I press Add and one that can handle exceptions: When I try to press Add that calls the same WhenIPressAdd() method but surrounded with a try/catch block and handling like you suggest. Now the system can throw errors to me, and I can catch and handle them on demand. – Louettalough

I do unit test at the business rules level which exceptions will be thrown and never caught. -- it will be caught at one level above. So your solution is very useful to me. Thanks! – Unable

@Kjetil Klaussen, is this approach still suitable nearly eight years on? +1. – Counterpressure

@Counterpressure yes, in this particular case it would be. But again; you really shouldn't write scenarios like this. It's an anti-pattern, because the scenenarios should be written with the end user in mind. And no end user would ever want an exception to be thrown... – Hasid

@Kjetil Klaussen, thanks. Are you saying that validation errors should not be "tested" for using Specflow? I do test them also using BDD. – Counterpressure

@Counterpressure No, by all means; validations should be tested. I just meant that you should test them according to what the end user should expect. The end result of an exception might be a validation error, which you should assert against in the scenario. – Hasid

Answer 3

BDD can be practiced on feature level behavior or/and on unit level behavior.

SpecFlow is a BDD tool that focuses on feature level behavior. Exceptions are not something that you should specify/observe on feature level behavior. Exceptions should be specified/observed on unit-level behavior.

Think of SpecFlow scenarios as a live specification for the non technical stakeholder. You would also not write in the specification that an exception is thrown, but how the system behaves in such a case.

If you do not have any non technical stakeholders, then SpecFlow is the wrong tool for you! Don't waste energy in creating business readable specifications if there is nobody interested in reading them!

There are BDD tools that focus on unit level behavior. In .NET the most popular one is MSpec (http://github.com/machine/machine.specifications). BDD on unit-level can also easily be practices with standard unit-testing frameworks.

That said, you could still check for an exception in SpecFlow.

Here are some more discussion of bdd on unit-level vs. bdd on feature-level: SpecFlow/BDD vs Unit Testing BDD for Acceptance Tests vs. BDD for Unit Tests (or: ATDD vs. TDD)

Also have look at this blog post: Classifying BDD Tools (Unit-Test-Driven vs. Acceptance Test Driven) and a bit of BDD history

Irresistible

I understand the distinction between ATDD and TDD as described in the blog post mentioned, but that leads me to a question. As described, isn't using a BDD tool (such as MSpec) just another unit testing framework? It seems to me that it is. Furthermore, if I can use the same tool for both ATDD and TDD, why shouldn't I? There seems to still be some blurry lines here. – Trocar

Hi Brian - most of the BDD tools are designed to help get a shared understanding with a non-technical stakeholder. There aren't so many BDD tools for a unit level / technical stakeholders, just because technical people can usually do unit-level BDD with TDD frameworks just fine. I use NUnit for both at the moment, with an English-style DSL underneath for scenarios. You can do that if it works for you. The only thing I do differently is to keep the scenario steps high-level so that I can reuse them - reuse is far greater than at a unit level. – Esquivel

Specflow is perfect to use even if everybody looking at the tests are pure technical. My team members and myself find tests written in Specflow so much intuitive and easy to write and re-use than others. They are NOT a waste of energy. – Vein

Answer 4

Changing the scenario not to have an exception is a probably good way to have the scenario more user oriented. However, if you still need to have it working, please consider the following:

1. Catch an exception (I really recommend catching specific exceptions unless you really need to catch all) in the step that invokes an operation and pass it to the scenario context.

   [When("I press add")]
   public void WhenIPressAdd()
   {
      try
      {
        _calc.Add();
      }
      catch (Exception err)
      {
         ScenarioContext.Current[("Error")] = err;
      }
   }
   

2. Validate that exception is stored in the scenario context

   [Then(@"it should throw an exception")]
   public void ThenItShouldThrowAnException()
   {
         Assert.IsTrue(ScenarioContext.Current.ContainsKey("Error"));
   }
   

P.S. It's very close to one of the existing answers. However, if you try getting value from ScenarioContext using syntax like below:

var err = ScenarioContext.Current["Error"]

it will throw another exception in case if "Error" key doesn't exist (and that will fail all scenarios that perform calculations with correct parameters). So ScenarioContext.Current.ContainsKey may be just more appropriate

Tat

Answer 5

My solution involves a couple of items to implement, but at the very end it will look much more elegant:

@CatchException
Scenario: Faulty operation throws exception
    Given Some Context
    When Some faulty operation invoked
    Then Exception thrown with type 'ValidationException' and message 'Validation failed'

To make this work, follow those 3 steps:

Step 1

Mark Scenarios you expect exceptions in with some tag, e.g. @CatchException:

@CatchException
Scenario: ...

Step 2

Define an AfterStep handler to change ScenarioContext.TestStatus to be OK. You may only want ignore errors in for When steps, so you can still fail a test in Then verifying an exception. Had to do this through reflection as TestStatus property is internal:

[AfterStep("CatchException")]
public void CatchException()
{
    if (ScenarioContext.Current.StepContext.StepInfo.StepDefinitionType == StepDefinitionType.When)
    {
        PropertyInfo testStatusProperty = typeof(ScenarioContext).GetProperty("TestStatus", BindingFlags.NonPublic | BindingFlags.Instance);
        testStatusProperty.SetValue(ScenarioContext.Current, TestStatus.OK);
    }
}

Step 3

Validate TestError the same way you would validate anything within ScenarioContext.

[Then(@"Exception thrown with type '(.*)' and message '(.*)'")]
public void ThenExceptionThrown(string type, string message)
{
    Assert.AreEqual(type, ScenarioContext.Current.TestError.GetType().Name);
    Assert.AreEqual(message, ScenarioContext.Current.TestError.Message);
}

Landa

In newer versions the TestStatus property changed to ScenarioExecutionStatus with a public setter (so less chance of breaking changes in the future), you can use it now as follows: PropertyInfo testStatusProperty = typeof(ScenarioContext).GetProperty(nameof(ScenarioContext.Current.ScenarioExecutionStatus), BindingFlags.Public | BindingFlags.Instance); testStatusProperty.SetValue(ScenarioContext.Current, ScenarioExecutionStatus.OK); – Ambur

Answer 6

In case you are testing user interactions I will only advice what has already been said about focusing on the user experience: "Then the user is presented with an error message". But, in case you are testing a level below the UI, I'd like to share my experience:

I'm using SpecFlow to develop a business layer. In my case, I don't care about the UI interactions, but I still find extremely useful the BDD approach and SpecFlow.

In the business layer I don't want specs that say "Then the user is presented with an error message", but actually verifying that the service correctly responds to a wrong input. I've done for a while what has already been said of catching the exception at the "When" and verifying it at the "Then", but I find this option not optimal, because if you reuse the "When" step you could swallow an exception where you didn't expect it.

Currently, I'm using explicit "Then" clauses, some times without the "When", this way:

Scenario: Adding with an empty stack causes an error
     Given I have entered nothing into the calculator
     Then adding causes an error X

This allows me to specifically code the action and the exception detection in one step. I can reuse it to test as many error cases as I want and it doesn't make me add unrelated code to the non failing "When" steps.

Animal

I'm new to BDD, but I really dislike the pattern of "do something in a When and throw it in the context, then read it out in the Then". I think this will be harder and harder to maintain as the number of specs grow, and as more of them are reused. I've begun doing what you describe above and so far I'm liking it. – Edelsten

Answer 7

ScenarioContext.Current is deprecated with the latest version of SpecFlow, it is now recommended to add a POCO to the steps test class in the constructor to store/retrieve context between steps, i.e.

public class ExceptionContext
{
    public Exception Exception { get; set; }
}

private ExceptionContext _context;

public TestSteps(ExceptionContext context)
{
    _context = context;
}

And in your [When] binding....

try
{
   // do something
}
catch (MyException ex)
{
    _context.Exception = ex;
}

In your [Then] binding, assert that _context.Exception is set and of the exception type you expected.

Traps