Pex users: what are your Impressions of Pex and Automated Exploratory Testing in general?

Question

Those of you who have used Pex, what do you think its advantages and disadvantages are of Pex as a tool?

Also, what do you think are the advantages and disadvantages of "Automated Exploratory Testing" in general, as a supplement to TDD/Unit Testing?

Snitch

How is it Exploratory Testing in QA meaning? It looks like it's nowhere near to Exploratory Testing in regards to ISTQB/ISEB/IEEE/ISO. Is it not a tool that is analyzing your code and generates Test Cases based on rules it has implemented in it? Sounds more like Expert System than Exploratory Testing - en.wikipedia.org/wiki/Exploratory_testing . – Seamaid

@Seamaid "Automated Exploratory Testing" was phraseology straight from MS (channel9.msdn.com/posts/briankel/…). It looks like they have backed away from that terminology since then. – Snitch

Answer 1

Pex lets your write parameterized unit tests. In that sense, it totally fits into the TDD/unit testing flow: write the test, have Pex 'explore' it, find some failing tests, fix the code, and so forth.

The big advantage is that you can express your tests for classes of inputs, not just a couple hard-coded values. This gives more expressiveness for writing tests and also forces to think about the invariant/expectation that your code should fullfill (i.e. it's harder to write assertions).

Friend

Edited the description to point out possible overlap, I wanted to get feedback as to how much of their unit tests TDDer's using Pex were able to parameterize. Also, I'd like to know the disadvantages as well as advantages. – Snitch

Answer 2

I think Pex as an exploratory testing tool is really intriguing. In that regard, I see it as something I'd want to hand off to QA to use.

As a TDD tool, it needs some work, as TDD is a design activity. However, I do like the direction that Peli is heading. There's something to be said for automated assisted design. For example, just because TDD is a design tool, there's no reason I can't have an automated tool point out potential edge cases while I'm designing, right? Build quality in from the start.

Check out this post in which Peli uses Pex in a TDD style workflow. http://blog.dotnetwiki.org/TDDingABinaryHeapWithPexPart1.aspx

Franko

is tdd really a design activity? I don;t think i have seen anyone let TDD make any design decisions for them. They design independantly and use TDD to develop code. – Midshipmite

I'm trying to get better at my TDD and finding that TDD really is a design activity, leading the design more than the actual code itself – Mccomas

Answer 3

If you look for literature on writing Theories (google David Saff) - which are a more general way of writing unit tests, and use Pex as a theory explorer I've found a step change in productivity from my experience so far. I've just wrote a blog post detailing my experiences of Pex in TDD, here: http://taumuon-jabuka.blogspot.com/2009/01/theory-driven-development-using_11.html

and as I said - I see it as TDD on steroids! It in no way replaces TDD, but enhances the activity.

Hukill

this is the way I m feeling about it at the moment, but I m trying to figure why Pex doesnt seem to be massive – Kerekes

Answer 4

I'm really pumped about Pex. It will provide tests for edege cases that you won't ever dream up, especially if your team is small and the person writing the methods is the same as the person writing the tests.

It will also provide contractual obligations that your methods will obey.

Softy

Answer 5

Test-first development makes you structure your code for testability. In this respect, Pex finds clever and awkward paths thru your code, helping out beyond simple coverage metrics.

Major forte of Pex with Moles is enabling of tracking side effects when doing Brownfield development: run Pex once and save outputs, then apply code changes, and run Pex again to see what got broken.

Dutchman