How can I perform mutation testing of my Java program?

C

7

Can anyone provide suggestions of tools that can be used to make mutations within a Java program at a source-code (not byte-code) level? I need to seed my source code with faults. I would prefer an application with a GUI, if one exists.

Cronyism answered 27/6, 2011 at 1:6 Comment(7)

This has little application in the real world – Collator 27/6, 2011 at 1:11

Simply replacing some random characters by others could do, but in most cases this will be letal (i.e. the resulting program will not be compilable, if you did't hit a string or comment or replaced really luckily). – Blowfish 27/6, 2011 at 1:11

"This has little application in the real world" is a poor excuse. When tensor, topology or calculus mathematics were developed, they had very "little real world use". This is an interesting question. Code mutation has very real world applications. – Nameless 27/6, 2011 at 1:32

It might be useful if you could provide suggestions (i.e. examples) of mutations you would like and the attributes of the result - should it still compile, should the errors be regularly placed, etc. – Undercut 27/6, 2011 at 1:40

@Bohemian: It is applied in the real world for determining test coverage. – Irrelative 27/6, 2011 at 1:55

How is this question too localized? Mutant (or mutation) testing is a valid testing technique that can be used to test software applications. However, it's not commonly used since it's time-intensive (and therefore, for most projects, cost-intensive as well). – Adin 27/6, 2011 at 2:5

pitest.org is the number 1 Mutation Testing tool for Java today (2023), but this is done at byte code level rather than source code level – Leilanileininger 11/3, 2023 at 14:12

C

4

I've found https://pitest.org, which I quite like. With MuJava I have exceptions caused by @Override annotations. Seems that it doesnt' support Java 5/6.

Cronyism answered 27/6, 2011 at 17:40 Comment(2)

Good overview of PIT here: dev.theladders.com/2013/02/… – Thaumatrope 5/3, 2013 at 15:25

Broken link above. Baeldung have a recent article (2022) baeldung.com/java-mutation-testing-with-pitest – Leilanileininger 11/3, 2023 at 14:25

S

4

If your requirement really is for source code mutation then the options I'm aware of are

Jester - Provides a limited number of mutation operators that are quite unstable. Jeff Offutt described it as a very expensive way to apply branch testing. I believe it is possible to define your own operators however, so this probably isn't entirely fair.

Mu Java - Doesn't support Java 5

Judy - Don't know much about this other than that it exists

Lava - Sounds pretty basic and I don't think it supports java 5

From the answer further down I see your now using PIT. I think that's a truly great choice (I wrote it so may have a certain bias), but it's a byte code mutation system.

Schilt answered 28/6, 2011 at 18:42 Comment(0)

A

3

When I took a software engineering class on testing at GMU, we used Mu Java, for one of the chapters. HTH

Ancylostomiasis answered 27/6, 2011 at 1:11 Comment(2)

Thanks! I also tried it, but it doesn't work for me. @Override annotation is causing exceptions.. – Cronyism 27/6, 2011 at 17:41

Since 2016, it appears muJava is now abandonware: github.com/jeffoffutt/muJava – Leilanileininger 11/3, 2023 at 14:33

I

3

This is the technique used by the open source test coverage tool Jester. A look at it's source might be helpful if you are unable to find a suitable tool and want to build one yourself.

Irrelative answered 27/6, 2011 at 2:1 Comment(1)

Since 2009, it appears Jester is now abandonware: sourceforge.net/projects/jester/files – Leilanileininger 11/3, 2023 at 14:32

Y

2

"This has little application in the real world". Disagree strongly, I contract at a very big media company in the UK and PIT (http://pitest.org/) is extremely useful for mutation testing. Most of the CI builds are set fail without 85% coverage PIT mutation testing. Re: "replacing some random characters..." Mutation does not have to mean that. The PIT mutations are restricted to compilable changes.

Youngran answered 12/12, 2013 at 17:34 Comment(1)

How do you deal with equivalent mutations if you require a 100% mutation coverage? – Schilt 15/12, 2013 at 20:5

G

1

Here it is a benchmark :

MμClipse only supports JUnit 3 and is no longer maintained. Jester as for it, is laborious and requires a complicated configuration; plus is not maintained anymore. The best tool I could find is Javalanche I had wrote a entire article about this !

Grazing answered 10/3, 2012 at 10:57 Comment(1)

Since 2013, it appears MuClipse is now abandonware: sourceforge.net/projects/muclipse (MuClipse was an Eclipse plugin wrapper for MuJava, which itself became abandonware in 2016) – Leilanileininger 11/3, 2023 at 14:30

O

1

The Major mutation framework (Major's website) provides a compiler-integrated mutator and a mutation analyzer for JUnit tests.

Major's mutator is integrated in the Java 7 compiler. It provides several mutation operators and supports two options for mutating source code:

Generate and embed all mutants during compilation
Generate mutants and export mutated source files

Ontologism answered 18/6, 2014 at 9:0 Comment(0)

Recommended topics

Hot tags