How to ensure quality of junit tests?

Question

Are there proven ways of verifying quality of junit tests or integration tests?

Should your business analyst review unit tests to cerfity? Or are there any other ways? In the traditional code first environment a peer or lead would review the test plan but how about automated tests?

I looked at this stackflow thread but couldn't extract anything meaningful stuff.

Thoughts?

Calends

Answer 1

Mutation testing and code coverage can verify the quality of your tests.

So first check than your code coverage is high enough. After this verify with mutation testing than your test are good. Mutation testing tool makes small change(s) in production code and reruns test - after a modification a good test should fail. For mutation testing tool in Java look at PIT Mutation Testing and this blog post: Introduction to mutation testing with PIT and TestNG

But this is still not enough, tests should be good written and readable. So you need code review also and quality rules verification for tests. I recommend nice book about writing good tests Practical Unit Testing. Chapter 10: Maintainable tests from this book is available for free.

Mormon

Here is a nice example of the mutation test way of thinking : wp.me/prMeE-11 – Entente

@Mormon can mutation testing ensure a hundred percent junit code quality? – Dramatic

Answer 2

Here's a nice linked article:

http://www.ibm.com/developerworks/java/library/j-cq01316/index.html?ca=drs

And:

Good Tests ⇒ High Coverage High Coverage ⇒/⇒ Good Tests

Coverage tools are useful to identify what areas of your project need more attention, but it doesn't mean that areas with good coverage shouldn't need more attention.

Catlike

Additional paper: ibm.com/developerworks/java/library/j-jester/j-jester-pdf.pdf – Zaragoza

Answer 3

Code coverage tool is a good start, but knowing that a given line was executed does not mean it was tested. Infamous test cases without assertions or expected=Exception.class are an example.

I can imagine few criteria on this level:

• if the line is tested, any change to it (inverting condition, removing...) should fail at least one test
• given piece of logic should be fully reconstructible based only its tests
• the test does not mirror the production code
• the test should not be dependent on current date, locale, timezone, order of other tests

One might try to automate the first one, others are more or less subjective.

As for analyst doing test review - probably only Fitensse fixtures are readable enough to satisfy non-developers.

Thatcher

Answer 4

Code review is the best way to ensure test quality. I would not have business analysts review the tests, for the simple fact that they might not have the training necessary to understand the tests. Also, unit tests do not all live at the functional level, where analysts' requirements are. An analyst might say 'when the user clicks save, the profile is saved' whereas you might have to write n number of tests across multiple layers to get that functionality.

Gerrigerrie

Code-review is a tool, but a tool is useless if the people using it are not aware of how best to use it. The people doing the code-review need to have the proper test-critical mindset to identify if something is tested adequately or not. Tools can provide useful metrics to help identify areas that need attention, but ultimately it's the person writing the code who knows best what the corner cases of his/her component are. – Gusti

Answer 5

You might consider code coverage tools to ensure 100% of the code lines are being tested. Emma is a good tool for java (http://emma.sourceforge.net/).

Danelaw

I wouldn't say that a 100% code coverage result ensures the quality of your JUnit tests. This could lead to developers adding some useless tests just to be sure their coverage result is enough. – Chopine

im not gonna downvote, but coverage is not a metric of quality, its a metric of coverage. Having 100% coverage does not ensure that the tests are testing anything. – Gerrigerrie

Code coverage will only ensure that every line of code has a tests. It still does not certify quality of my tests in a true TDD or Test First environment. Did I miss anything? – Calends

I agree that it's not the best measure of quality, but it does provide a metric as to how much unit tests are required or expected. In the end, the quality of your unit tests depends on the quality of your hiring process (and the degree to which management subsequently holds developers accountable for writing tests). Code coverage is one (measurable) way to enforce that a certain minimum amount of tests need to be written. – Danelaw