What mutation-testing frameworks exist? [closed]

P

11

32

In another question I asked if mutation-testing is useful in practice. As I didn't get any answers that satisfy me, I want to check the current tools myself. So I need an overview about current existing mutation-test-frameworks. Which of them are most usable and why?

I program in Java, so I would prefer Java-tools, but I would risk a look at interesting frameworks for different languages.
I want to integrate in an automatic build-process, so I would prefer tools that can be executed through command-line.

Perigon answered 29/10, 2008 at 11:47 Comment(1)

An explanation of the advantages of MT: pedrorijo.com/blog/intro-mutation – Lyford 15/2, 2019 at 9:28

L

17

There is also PIT which can be hooked into your build via a maven plugin or command line interface.

It provides much nicer reports than the other available tools with combined mutation and line coverage. It also runs considerably faster than the source based tools for Java such as Jester, and about twice as fast as Jumble.

Unlike the Jumble and Javalanche it also works with all the major mocking frameworks (Mockito, JMock, EasyMock, PowerMock and JMockit).

(disclosure I'm the author).

Latea answered 1/3, 2011 at 22:30 Comment(2)

Is there anyway I can save the PIT generated mutants to my desired folder? so that can be used by other programs? – Myranda 15/1, 2015 at 22:15

@Myranda nothing built in, but if you check the google group history you'll find posts describing how to hook into the framework and write them to disk – Latea 16/1, 2015 at 9:48

M

16

I know it's an old thread, but it's still an answer to the question. I'm working with some friends on an open source .NET mutation testing framework called NinjaTurtles, which you can find on CodePlex and on Nuget. The main project website is here.

Milieu answered 23/4, 2012 at 14:45 Comment(2)

I'vre tried the NinjaTurtles, but it is badly documented and i can't figure out what's wrong or what exactly am i doing wrong... So i rather tried CREAM and it works nicely. – Viviparous 14/1, 2014 at 12:7

Is it still being developed, I am willing to help. – Category 15/2, 2018 at 2:16

P

10

I only know of two frameworks, but they're both for Java :)

I haven't used either of them, I'm afraid.

Pernas answered 29/10, 2008 at 12:3 Comment(1)

The link for Jester has an extra p://. – Escarpment 7/6, 2010 at 13:49

S

10

CREAM is a tool for C#/.Net http://galera.ii.pw.edu.pl/~adr/CREAM/index.php

Squash answered 15/6, 2011 at 19:29 Comment(1)

This tool looks very good and supports .NET 4.0 etc. Thanks +1 – Wellchosen 6/9, 2011 at 23:28

A

5

For Ruby there is Heckle, and a newcomer called Boo_hiss.

Atbara answered 13/2, 2009 at 11:17 Comment(2)

<Spam>Check out chaser. It is the new awesomeness for ruby.</Spam> – Escarpment 14/11, 2009 at 3:52

<Spam>And zombie-chaser. Mutation testing ... with zombies!</Spam> – Escarpment 6/5, 2010 at 4:3

T

3

For the .Net community, there is NesTer, but it has some serious limitations. E.g. only supports C# and NUnit.

Does not appear to be actively maintained either, but it might be a starting point.

Tribalism answered 15/4, 2009 at 9:35 Comment(1)

Only works for solutions developed in visual studio 2005 – Mattland 13/5, 2016 at 10:19

H

3

I took a look at Jester (the actual source code) and it seems to me that it does not support too many mutations. There is a file in there where these mutations are specified. I might be wrong about the above but what I definitely did not like was the mix between launching the tool from command line and the little GUI feedback interface. Why not give feedback in the command line like JUnit does when run outside an IDE?

Jumble is another thing :). It has a simple command line interface and comes with an Eclipse plugin too. The feedback is all text in the console. I am happy with this tool and I plan to write some ANT target to add it in my project continuous integration.

I am also looking at Javalanche but did not try it yet.

I'll have news in a few weeks.

Heptachord answered 28/1, 2010 at 7:56 Comment(0)

R

3

Might be of some interest. Microsoft Research's: https://pex4fun.com/

Roentgenoscope answered 22/10, 2012 at 13:46 Comment(2)

@urig thanks for highlighting, URL updated. – Roentgenoscope 25/1, 2018 at 12:13

Thanks for the new URL @nashwan. I deleted my comment so as not to mislead future readers. – Diva 25/1, 2018 at 14:29

V

2

You can try µJava. I haven't used it, but it looks like mutation testing might be an interesting way to evaluate test suites.

Veranda answered 29/10, 2008 at 12:13 Comment(0)

M

2

MμClipse only supports JUnit 3 and is no longer maintained. Jester as for it, is laborious and requires a complicated configuration; plus is not maintained anymore. The best tool I could find is Javalanche I had wrote a entire article about this !

Megacycle answered 10/3, 2012 at 10:41 Comment(0)

M

0

Jester does provide a file for the mutations and they are limited. To some degree, you can add your own mutations to the file.

I've experimented with Jumble and Jester and I found that Jumble provides more mutations and better documentation. Additionally, I've had quick responses from the project owners when I've emailed them. One drawback to Jumble is that it operates on the bytecode using BCEL. That presents something of a learning curve for many developers.

My company, State Farm, wrote an Ant task that we may contribute back to the Jumble project. Based on what I've read in their mailing lists, others are working on an Ant task for Jumble too.

I'm looking at Javalanche as well. I’ll be glad to share what I know when I’m done.

Mays answered 15/2, 2010 at 15:42 Comment(0)

Recommended topics

Hot tags