Is Pex (Test generation) really useful tool?

Question

Yes, it is possible to generate tests on boundary values for functions like "Sum" or "Divide". Pex is a good tool here.

But more often we create tests on business behaviour. Let's consider example from classic Beck's tdd book:

[Test]
public void ShouldRoundOnCreation()
{
  Money money = new Money(20.678);
  Assert.AreEqual(20.68,money.Amount);
  Assert.AreEqual(2068,money.Cents);
}

Can this test be generated? No :) 95 % of tests in my projects check business logic, and can not be generated.

Pex (Especially in pair with Moles) can give 100% code coverage, but a high code coverage rate of a test suite does never indicate, that code is well tested - It only gives false confidence that everything is tested. And this is very dangerous.

So, the question is - Is Pex really useful tool?

Danyel

Answer 1

I think you are mistaken in the way Pex should be used: we highly recommend users to write assertions in their parameterized unit tests.

If you write assertions, Pex will try systematically to invalidate them - that's where the power of Pex comes in: the more you write assertions, the more it tries to find bugs for you.

If you use Pex to get a high code coverage test suite without writing assertions, you only get what you asked for: code coverage and guaranteed runtime exceptions. Pex 'only' tries to cover branches (1 assertion = 1 branch), if there are no branches to cover (no assertion), it won't generate interresting test cases.

In general, writing assertions in parameterized unit tests are harder to write because... well they are more general. Let's start with the rounding behavior: there is certainly a bound on which you allow rounding to occur, this should translate naturally into a parameterized unit test:

[PexMethod]
public void RoundInBounds(double value) 
{
    var money = new Money(value);
    // distance between value and amount should be at most 0.01/2
    Assert.AreEqual(value, money.Amount, 0.005);
}

There are also a number of patterns that can be used to write those. For example, your 'Money' class is probably idempotent: if you feed the value of 'Amount' back in a Money instance, you get Amount back. This translate elegantly into a parameterized unit test:

[PexMethod]
public void RoundIsIdempotent(double value) 
{
     var first = new Money(value).Amount;
     var second = new Money(first).Amount;
     Assert.AreEqual(first, second, 0.0001);
}

Note also, that parameterized unit tests definitely belong in the TDD world. Just write the parameterized unit test first, Pex will find the failing bug, fix the bug, Pex finds the next failing bug, etc...

Does this make Pex a useful tool? You be the judge.

Jurisprudence

To make story short: 1) Pex can not generate sufficient tests by iteself. 2) It can be used to generate parameters for parametrized tests. The second one is usefull, agreed. But what about regression? If I find that the code snippet fails with parameter = 123,43, probably I would like to save somewhere this case probably in nunit parametrized test. Have you ever tried to mix pex and nunit parametrized tests? – Danyel

Out of curiosity, why are using NUnit when you're using Visual Studio? – Alecalecia

I like NUnit more than MSTest because it more complete. However I really do not have anything against MSTest. – Danyel

You are welcome to mix traditional unit tests and parameterized unit tests: parameterize unit tests are simply method with parameters. If you need to add a regression scenario, just add it - don't get stuck in one methology, embrace all. – Jurisprudence

Pex can generate unit tests for MSTest, NUnit, xUnit.net or MbUnit out of the box. In fact, it is extendible so you can come up with your own unit test framework as well and integrate it. – Jurisprudence

This is a good answer. It is not a framework's fault if naive developers incorrectly use it en masse. They're just as likely to adopt some other library incorrectly as well. A tool is just that, not some kind of silver bullet. – Talkie

Answer 2

There's a few things useful in Pex.

1. Code coverage. Let's put Pex aside for a second. In general, 100% code coverage doesn't necessarily mean you have good code coverage. It just means ever path is executed, but program has data-flow, and testing that code different, additional inputs, gives you best "test coverage", if not code coverage. Here's, I'm just reiterating your point. 100% code coverage is not necessarily good, but you can say for sure that 25% code coverage is bad, so that's how code coverage is useful. When you have low code coverage, you know for sure you have low test coverage.

   When you use Pex to get 100% code coverage, it's not really helping you get better test coverage per se, but what it does do is give ever piece of the production code some test, which can be used for debugger. In fact, a Pex presentation as a conference shows the use of Pex for this very purpose. The programmer said, "Gee, look at this method in NHibernate. I'd like to step through it in a debugger to see what it does, but how do I even invoke that method through the normal "business-entry point" into the library? Without knowing anything about the library, you can't. So he ran Pex, and was able to step through the code with all sorts of parameters. Interesting, Yes. Useful, maybe so, maybe no.

2. In addition to automatically created tests, Pex is also useful for parameterizing your tests. It's much better to create unit tests that are data driven. Why write the same code over and over, with different parameters. Write it once, and feed the parameters from a data source.

3. Pex is also useful as a simple stubbing framework. It's probably the easiest way to create mock objects, using the new lambda expression, which is much easier to understand than complex frameworks like RhinoMocks, etc. With Moles, you can also stub not just interface, but concrete non-virtual methods in classes.

4. I would also be careful with the "testing at the business logic" layer, too much. You could easy get to doing Behavioral Driven Development, which is not for unit testing. There, you are testing against a spec. If that's all you did, how would you test, say, custom data structures, etc., that have no business value, but are internal libraries used throughout the application.

Alecalecia

1) Yas 100% coverage is not necessarily mean good and that is why it is dangerous. 2) You can use classic tests to run code under debugger. But if you have proper tests, it is very rare case when you need a debugger. 3) Nunit also supports paramerized tests, the only difference - parameters are not generated. 4) Moles can be used separately from Pex. The only advantage of Moles in front of Rhino or Moq is that it can create stubs for static calls and contructors (and this is not good from design prospective). 5) BDD - is just subset of TDD, done in slightly different way. – Danyel

be careful saying it's dangerous. just because it can be misused doesn't mean it can't also be useful. BDD is not a subset of TDD -- BDD is at a higher level, capturing user stories, behavior, where as unit tests capture structure and design of the code. – Alecalecia

I did not say that Pex is useless and dangerous because it can provide 100% coverage. I just said that 100% of coverage gives false confidence. Regarding BDD - Don't you think that it is possible to capture user stories in regular test (unit, integration and functional)? (code-magazine.com/article.aspx?quickid=0805061&page=1 the last paragraph) – Danyel

This answer is based on false premise after false premise. Not looking to argue, just pointing it out. – Talkie

Point 3 is not related to PEX and should be removed from the answer. Moles have been replaced by Fakes in VS 2012 – Nb