What mutation-testing frameworks exist for C/C++?

Question

Mutation testing has been out there for a while now, and it seems there are at least one or two commercial mutation testing frameworks for C/C++. Have you used them? What are your experiences? Are there any open source alternatives?

Halliday

Good question as I did not know about mutation-testing. Great idea ^^ – Silicious

See #246995 – Polypoid

Answer 1

A brief search resulted in:

• PlexTest: http://www.itregister.com.au/products/plextest_detail.htm
• Insure++: http://www.parasoft.com/jsp/products/insure.jsp;jsessionid=baacpvbaDywLID?itemId=63
• MILU (may be only for C): http://www.dcs.kcl.ac.uk/pg/jiayue/milu/

With that said, you need to realize that mutation testing isn't particularly useful (at least from some stuff I've previously read). It's an interesting tool when faced with hard (metaphorically speaking) asserts and for making sure that data requirements are heeded to (when dealing with if and only if situations).

In my opinion, there are much more established ways of analyzing the robustness of code.

Proa

What Parasoft (Insure++) calls "mutation testing" is not much like what everyone else calls mutation testing. They "revolutionized" it beyond recognition. parasoft.com/jsp/products/article.jsp?articleId=291 – Halliday

@Jason: it seems like an interesting take on it, but the idea is still the same.. take the code, change some operators (either making them "faulty" or "equivalent"), and see if stuff works. Finding ambiguities in code is sometimes useful, but to be completely honest, you could write your own mutation-testing tool (it's not hard) and save some money. – Proa

This is the first result that pops up on google, so I thought it might still be relevant to point out that the MILU link leads to a 404. – Archery

I really think the statement "mutation testing isn't particularly useful" should either be qualified, or outright removed from the answer. – Dissipation

It is easy to find popular tools for Java. Why is it hard to find popular tools for C++? In fact I love C++ (I love Java too) and want to do some serious C++ development and I am hunting for tools that can help me write good quality code. We have tools such as Eclemma, Pitest, SonarLint, JProfiler, VisualVM, etc... for Java. Could anyone please give such popular and proven tool(s) for C++ please? I want expert advise here. – Circe

All the 3 links are DEAD. Plz update – Immoderacy

Answer 2

Notice that Parasoft's tool only generate equivalent mutations. That echoes the problem described on Wikipedia article about Mutation Testing - it is hard to distinguish between equivalent and non-equivalent mutations so they decided to stick with equivalent.

I tried another interesting tool that can automatically discover invariants in instrumented C and C++ code - it is called "Daikon". Essentially it is doing same thing as tool that generates equivalent mutations, but instead of identifying problematic code it gives you a set of invariants such as "A == B + 1". I think invariants are more useful because when you look at discovered invariant it gives you assurance that your code is correct if invariant make sense, and then you can convert invariants into asserts and that gives you more confidence when you change code.

Hohenlohe

Answer 3

A straight forward python script for mutating c programs is available at:

https://github.com/parunbabu/mutate.py

the author says it works better if the code under test is de-commented and indented.

and it is also free and opensource ... i think this is what you are looking for.

Ermine

Answer 4

• Mull is LLVM-based and seems to be actively developed and easy to use.
• dextool mutate also LLVM-based and actively developed, more complicated to use but has more features like re-running alive mutants and only mutate introduced changes based on a git diff

Thrave

Answer 5

I have no experience with it but Mutate++ seems to be an option that is missing from the ones already mentioned.

Mutate++ - A C++ Mutation Test Environment

Enchantment

Answer 6

The existing frameworks where way too time-consuming to set up and use so I did my own implementation, a quick and easy solution that should work on any machine. There is binaries available for MacOSX, Windows and RaspberryPi (Linux):

https://github.com/RagnarDa/dumbmutate

Hope it helps anyone!

Diehl