Android: How to run PIT Mutation Testing with Robolectric?

How to use Robolectric and PIT for testing an Android Application?

With Robolectric, you can run Android Tests in the JVM. With PIT you can show line coverage and do mutation testing. For me, it's ok use Eclipse+Plugins, but no requirement.

This is what I have tried so far:

I have an Android Project, let's call it MyProject.

I now want to test MyProject in the JVM using Robolectric and PIT. Therefore, I created another Project called MyTest and managed to run Robolectric tests successfully, just as described in the robolectric quick start. This is what my.app.tests.MyActivityTest looks like:

@RunWith(RobolectricTestRunner.class)
public class MyActivityTest {
    @Test
    public void myTest() throws Exception {
        String appName = new MainActivity().getResources().getString(R.string.app_name);
        Assert.assertEquals(appName, "MyProject");
    }
}

Now the tricky part: I want to add PIT's Line Coverage and Mutation Testing to my Robolectric tests. First tried to use Pitclipse - had no luck. Pitclipse doesn't seem to support Eclipse Project Depencies yet.

So my second try is using the command line, as described in PIT quick start:

First, I made sure my tests run through successfully using Junit from command line:

java -cp <classpath> org.junit.runner.JUnitCore my.app.tests.MyActivityTest

The <classpath> contains: junit4, robolectric, MyProject class files, MyTest class files, android.jar, and other necessary android libraries.

Once this JUnit test was successful, I used the same <classpath> in my PIT call, and I execute that call in the root path of MyProject:

java -cp ../MyTest/bin:../MyTest/libs/*:bin/classes:~/android-sdk-linux/platforms/android-17/android.jar \
    org.pitest.mutationtest.MutationCoverageReport \
    --reportDir ../MyTest/pit-report \
    --targetClasses my.app.* \      # package in MyProject
    --targetTests my.app.tests.* \  # package in MyTest
    --sourceDirs src/

However, this results in the Exception I posted below. I think I need to exclude some classes using PIT's --excludedClasses parameter, but there is no hint about which class might cause the trouble. Note that MyActivityTest has no super class and no explicit constructor.

java.lang.NullPointerException
ERROR Description [testClass=my.app.tests.MyActivityTest, name=myTest(my.app.tests.MyActivityTest)] -> java.lang.NullPointerException
    at org.pitest.boot.CodeCoverageStore.visitProbes(CodeCoverageStore.java:92)
    at my.app.tests.MyActivityTest.<init>(MyActivityTest.java:22)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
    at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195)
    at org.robolectric.RobolectricTestRunner$HelperTestRunner.createTest(RobolectricTestRunner.java:647)
    at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241)
    at org.robolectric.RobolectricTestRunner$HelperTestRunner.methodBlock(RobolectricTestRunner.java:657)
    at org.robolectric.RobolectricTestRunner$2.evaluate(RobolectricTestRunner.java:227)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.robolectric.RobolectricTestRunner$1.evaluate(RobolectricTestRunner.java:175)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.pitest.junit.adapter.CustomRunnerExecutor.run(CustomRunnerExecutor.java:42)
    at org.pitest.junit.adapter.AdaptedJUnitTestUnit.execute(AdaptedJUnitTestUnit.java:86)
    at org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:50)
    at org.pitest.containers.UnContainer.submit(UnContainer.java:46)
    at org.pitest.Pitest$3.run(Pitest.java:148)
    at java.lang.Thread.run(Thread.java:679)

What looks to be happening is that two copies of pit's code coverage store class are being loaded. This is a class that tracks line coverage for each class against each test.

The classes under test are identified by an integer id that is assigned to them as they load - this id is embedded into probe calls added by bytecode manipulation that call out to the code coverage store class.

The code assumes that there will be an available entry in the store for each class id as each id is registered with the store on loading. This assumption is broken as the version of the class receiving the probe calls is different from the one against which the classes were originally registered.

This is a long way of saying that pit 0.31 and below do not look to be compatible with Roboelectric.

I'll need to take a look to see precisely what Roboelectric is doing behind the scenes to see if this can be fixed in a future release.

---- Update ---

0.32-SNAPSHOT release appears to work with Roboelectric (see comments).

Recommended topics

Hot tags