Is there hard evidence of the ROI of unit testing?

Question

Unit testing sounds great to me, but I'm not sure I should spend any time really learning it unless I can convince others that is has significant value. I have to convince the other programmers and, more importantly, the bean-counters in management, that all the extra time spent learning the testing framework, writing tests, keeping them updated, etc.. will pay for itself, and then some.

What proof is there? Has anyone actually developed the same software with two separate teams, one using unit testing and the other not, and compared the results? I doubt it. Am I just supposed to justify it with, "Look it up on the Internet, everybody's talking about it, so it must be the right thing to do"?

Where is the hard evidence that will convince the laymen that unit testing is worth the effort?

Singlecross

Answer 1

Yes. This is a link to a study by Boby George and Laurie Williams at NCST and a another by Nagappan et al. I'm sure there are more. Dr. Williams publications on testing may provide a good starting point for finding them.

[EDIT] The two papers above specifically reference TDD and show 15-35% increase in initial development time after adopting TDD, but a 40-90% decrease in pre-release defects. If you can't get at the full text versions, I suggest using Google Scholar to see if you can find a publicly available version.

Frame

The first study compares agile+TDD against waterfall projects, it's results would be more relevant if it had compared two agile teams. The second study mentions other studies that found little to no quality bonus for TDD projects. And when you compare management's estimates about needed extra time for TDD it's significantly estimated higher for the two teams with a high domain expertise, yet they also have a 20% lower test coverage. This confirms my own experience, I find assurance much more important in systems I haven't worked with yet, while testing is a hindrance for everything else. – Finicky

Neither of the studies compares comparable process model with only the testmethofology change. That is is spending the time used on UT actually better spent on eg. system testing. As it stands it might as well be " if we test smarter does that help" study. – Tingle

So what if the cost of fixing the post release bugs is 0.01% of total development? TDD would be a terrible investment in that case. And if the bugs are few? These %s mean nothing without context. To be fair I am yet to read the whole study. But as it stands your post is useful (good links) but does not answer the question regarding ROI, IMO. – Poem

@Poem Luckily (?) there is good evidence that this is not the case. Fixing post-release bugs is exponentially more expensive than bugs found early in development (which is what TDD does). In that context, a cost of 0.01% of total development for all post-release bugs seems unlikely. (For details see Code Complete, in particular Boehm & al., “Understanding and Controlling Software Costs”, IEEE Trans Softw Eng (1988)). – Keel

It's probably worth noting that the first study has a sample size of 24 programmers (working in pairs, so 12 teams). I'm not sure what a statistically valid sample size would be, but these seem low. Perhaps someone else knows? – Landel

Answer 2

" I have to convice the other programmers and, more importantly, the bean-counters in management, that all the extra time spent learning the testing framework, writing tests, keeping them updated, etc.. will pay for itself, and then some."

Why?

Why not just do it, quietly and discretely. You don't have to do it all at once. You can do this in little tiny pieces.

The framework learning takes very little time.

Writing one test, just one, takes very little time.

Without unit testing, all you have is some confidence in your software. With one unit test, you still have your confidence, plus proof that at least one test passes.

That's all it takes. No one needs to know you're doing it. Just do it.

Fauman

The bean counters couldn't tell a unit test from the rest of the code if their lives depended on it. I support the suggestion to just do it. There's one caveat, though: If you are not alone, you need your fellow developers to embrace this practice. If not, they will unintentionally break your tests. – Internationale

Just do it and don't tell them, and sell the idea to your colleges at the coffee break ;-) – Pinchhit

Well... they could say that "even if unit test works for you but it's not working for me because of my projects are more difficult than you" – Broadleaf

@Ngu Soon Hui: Not only will they say that, but they'll also say the opposite: "even if unit test works for you, I don't need it because my projects are simpler than yours". Folks will make up a ton of reasons to avoid "work" doing more work coding, debugging and fixing to avoid work of writing a unit test. "Convincing" through anything but a successful example is hard. – Fauman

Because you'd get fired when you consistently didn't hit your deadlines? – Roofing

Hi - good comments. And you never hear one of the developers sitting near you talking "&%^#(#)$&#( This Worked Yesterday!". If they had unit tests they would know exactly where it broke and why. We never have time to do it right, but always have time to do it again, and again, and... :) regards! – Avail

If the bean counters refuse to introduce Unit Tests, tell them to also fire all middle management on the project. I mean, if they don't care enough to add a bit of overhead in order to make sure that the code keeps on working, why are they will to pay such overhead to managers to make sure the employees are still working? "Hey, that guy worked productively a year ago, he's probably still working productively now." Perhaps the difference is that they mistakenly believe code is static, when in fact it evolves and changes, just like people. – Leclerc

@Neko: Unit tests don't add a "bit of overhead". They reduce the overall workload by preventing a whole flood of dumb mistakes. The work does not grow; it simply shifts in nature from bad code to good unit tests and good code. – Fauman

The bean counters want their engineers to provide sound solutions to the domain problems. You can just write tests as part of your solution. They won't even notice. If they ask you can just tell them you are spending more time on it to make sure it is robust and won't require reworking. If you SUGGEST writing unit tests to them you are asking their approval on something they know nothing about. – Houseless

Answer 3

I take a different approach to this:

What assurance do you have that your code is correct? Or that it doesn't break assumption X when someone on your team changes func1()? Without unit tests keeping you 'honest', I'm not sure you have much assurance.

The notion of keeping tests updated is interesting. The tests themselves don't often have to change. I've got 3x the test code compared to the production code, and the test code has been changed very little. It is, however, what lets me sleep well at night and the thing that allows me to tell the customer that I have confidence that I can implement the Y functionality without breaking the system.

Perhaps in academia there is evidence, but I've never worked anywhere in the commercial world where anyone would pay for such a test. I can tell you, however, that it has worked well for me, took little time to get accustomed to the testing framework and writing test made me really think about my requirements and the design, far more than I ever did when working on teams that wrote no tests.

Here's where it pays for itself: 1) You have confidence in your code and 2) You catch problems earlier than you would otherwise. You don't have the QA guy say "hey, you didn't bother bounds-checking the xyz() function, did you? He doesn't get to find that bug because you found it a month ago. That is good for him, good for you, good for the company and good for the customer.

Clearly this is anecdotal, but it has worked wonders for me. Not sure I can provide you with spreadsheets, but my customer is happy and that is the end goal.

Legibility

My QA guy was pretty sharp but he wasn't looking at code, but it was easy to tell the bounds weren't checked. – Legibility

Totally agreed about unit testing forcing you to think more about your design and correctness rather than code recklessly – Balsamic

Customers don't pay us to write tests. Then again, they don't pay us to write code, either. They pay us to solve their problems, and when confronted, I bet they also want the problems to stay solved. Given the evidence, it's unbelievable customers don't want to secure their investment. – Internationale

Answer 4

We've demonstrated with hard evidence that it's possible to write crappy software without Unit Testing. I believe there's even evidence for crappy software with Unit Testing. But this is not the point.

Unit Testing or Test Driven Development (TDD) is a Design technique, not a test technique. Code that's written test driven looks completely different from code that is not.

Even though this is not your question, I wonder if it's really the easiest way to go down the road and answer questions (and bring evidence that might be challenged by other reports) that might be asked wrong. Even if you find hard evidence for your case - somebody else might find hard evidence against.

Is it the business of the bean counters to determine how the technical people should work? Are they providing the cheapest tools in all cases because they believe you don't need more expensive ones?

This argument is either won based on trust (one of the fundamental values of agile teams) or lost based on role power of the winning party. Even if the TDD-proponents win based on role power I'd count it as lost.

Hippo

hear, hear :) A lot of the hard evidence for TDD also comes from very experienced teams that were already getting good results without it. TDD just improved their results rather than creating them out of thin air. The real ROI is hiring decent coders and letting them decide how to do things. – Communicative

"Is it the business of the bean counters to determine how the technical people should work?" --> all business decisions come down to money. Still, good answer, +1 – Corey

@Corey but how you perform your job has nothing to do with money and if you wish dome one to be accountable you let them decide HOW they do the WHAT you asked of them – Tingle

TDD is not a design technique, it is just a coding technique. blog.ploeh.dk/2010/12/22/TheTDDApostate Many commenters disagree that TDD involves refactoring (which is a design technique) but refactoring does not imply TDD. One can refactor without tests, large complex refactoring affects unit tests anyway i.e. tests need to be refactored too so can become invalid / false green just as well; simpler refactorings many not affect tests but the risk of error is lower - because refactoring is simple. – Sifuentes

@Sifuentes well, with the reflection of 10.5 years after this answer, I might phrase it a bit more defensive today, but still: I don't argue that TDD is the only design technique you'll ever need and Mark opens with it being a good design technique before concluding that it isn't one at all. I'd weaken his opinion and say it must not be the only design technique. Every code that I've ever written TDD looks different from code that I've written without. I'd call that a result of design. I work best with whiteboard, discussions, and other tools, in addition to TDD. But thanks for the link – Hippo

Answer 5

More about TDD than strictly unit testing, here is a link to the Realizing quality improvement through test driven development: results and experiences of four industrial teams paper, by Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. paper published by Microsoft Empirical Software Engineering and Measurement (ESM) group and already mentionned here.

The team found was that the TDD teams produced code that is between 60% and 90% percent better (in terms of defect density) than non-TDD teams. However TDD teams took between 15% and 35% longer to complete their projects.

Ninnyhammer

Answer 6

If you are also interested in evidence against unit testing here is one well researched and thought out article:

Why Most Unit Testing is Waste By James O Coplien (lean and agile guru)

Rodarte

Answer 7

Here's a great and entertaining read of a guy changing his company from within. It's not limited to TDD. http://jamesshore.com/Change-Diary/ Note that he didn't persuade the "bean counters" for quite some time and did "guerilla tactics" instead.

Censorship

the link looks interesting... worth checking out re: changing organisations work processes... – Defroster

Answer 8

Just to add more information to these answers, there are two meta-analysis resources that may help out figuring out productivity & quality effects on academic and industry background:

Guest Editors' Introduction: TDD—The Art of Fearless Programming [link]

All researchers seem to agree that TDD encourages better task focus and test coverage. The mere fact of more tests doesn't necessarily mean that software quality will be better, but the increased programmer attention to test design is nevertheless encouraging. If we view testing as sampling a very large population of potential behaviors, more tests mean a more thorough sample. To the extent that each test can find an important problem that none of the others can find, the tests are useful, especially if you can run them cheaply.

Table 1. A summary of selected empirical studies of test-driven development: industry participants*

Table 2. A summary of selected empirical studies of TDD: academic participants*

The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis [link]

Abstract:

This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity.

The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity. However, subgroup analysis has found both the quality improvement and the productivity drop to be much larger in industrial studies in comparison with academic studies. A larger drop of productivity was found in studies where the difference in test effort between the TDD and the control group's process was significant. A larger improvement in quality was also found in the academic studies when the difference in test effort is substantial; however, no conclusion could be derived regarding the industrial studies due to the lack of data.

Finally, the influence of developer experience and task size as moderator variables was investigated, and a statistically significant positive correlation was found between task size and the magnitude of the improvement in quality.

Demirelief

Answer 9

There are statistics that prove that fixing a bug found in the unit/integration test costs many times less than fixing once it's on the live system (they are based on monitoring thousand of real life projects).

Edit: for example, as pointed out, the book "Code Complete" reports on such studies (paragraph 20.3, "Relative Effectiveness of Quality Techniques"). But there is also private research in the consulting field that proves that as well.

Morry

This is covered in Steve McConnell's Code Complete, which is a book you probably want to have on your bookshelf for other reasons. – Evaporimeter

That's not related to the test method but to when in the process a bug is reported and further the time would be better spent finding bugs in the specifications since the cost of fixing them when finding them when developing is reported upto 1000 fold as expensive ( a factor of 10 per development phase) – Tingle

OTOH, if you only fix the problems that people actually encounter in real-life situations, you probably end up having to fix far fewer bugs. It's also not clear to me that fixing bugs earlier is really cheaper, since detecting a bug in a specification might require much more effort than detecting the same bug in implementation, and detecting the bug is part of the cost of the bugfix. This is one of these things that everybody just believes because it sounds self-evident, but I've never seen a sound study that showed the effect. – Vivianne

Answer 10

Well, there are some large companies that require you to use unit testing but if you are a small company why mimic large ones?

For me when I started with unit testing , many years ago,(today we mostly use behavior model) it was because I could not control all the path in one application.

I was used to bottom first programming and a REPL so when I got Unit Test (One Test for Every Function) it was like bringing back a REPL to languages that where very much compile. It brought the fun back to every line of code I wrote. I felt god. I liked it. I didn't need a report to tell me that I begun writing better code faster. My boss didn't need a report to notice that because we where doing crazy stuff we suddenly never missed a deadline. My boss didn't need a report to notice that the number of "plain" bugs drop from (to many) to nearly nil because of this very strange thing of writing non-productive code.

As another poster already wrote, you don't use TDD to Test (verify). You write it to capture the specification, the behaviour of what your unit(object, module, function, class, server, cluster) works.

There are lot of failures and success stories of switching to a different model of developing software in a lot of companies.

I just started to use it whenever I had something new to write. There is a old saying that goes somewhat hard for me to translate to english but:

Start with something so simple that you don't notice that you do it. When training for a marathon, start by walking 9 meters and run 1 meter, repeat.

Turbit

So, I should just do it? It's guaranteed to work, and it doesn't matter if no one else does it with me? – Singlecross

Actually, this is a Joel Test: joelonsoftware.com/articles/fog0000000043.html. It sounds to me that you may have more problem than a lack of the Nobel Prize Award Study On Unit Test – Turbit

Answer 11

I do have one set of data points for this - from an experience that sold me on unit tests.

Many moons ago I was a fresh graduate working on a large VB6 project and had occasion to write a large body of stored procedure code. Of the subsystem I was writing it made up about 1/4 of the whole code base - around 13,000 LOC out of 50K or so.

I wrote a set of unit tests for the stored procedures but unit testing VB6 UI code is not really feasible without tools like Rational Robot; at least it wasn't back then.

The statistics from QA on the piece were that about 40 or 50 defects were raised on the whole subsystem, of which two originated from the stored procedures. That's one defect per 6,500 lines of code vs. 1 per 1,000-1,200 or so across the whole piece. Bear in mind also, that about 2/3 of the VB6 code was boilerplate code for error handling and logging, identical across all of the procedures.

Without too much handwaving you can ascribe at least an order-of-magnitude improvement in defect rates to the unit testing.

Bitterweed