What are my alternatives for test code coverage on Android?

Question

I know Android's Ant tooling comes with built-in Emma code coverage, but Emma's reports look more and more dated and cannot be easily integrated into other coverage reports or even Sonar, basically because Emma lacks a detailed XML reporting format. What I want to achieve is

• code coverage for unit tests (Robolectric-based, already done with Cobertura)
• code coverage for integration tests (Robotium-/emulator-based, currently done separately with Emma)
• ideally merging code coverage results (this is IMHO only possible with Cobertura and requires the raw *.ser files from every test execution)
• publishing to Sonar (currently only done with the unit test coverage, Sonar understands only one coverage format to my knowledge)

Do I have any alternatives?

edit: This post on sonar-devel probably sums up the current state of Android code coverage support as well as Sonar support pretty much. sigh

Rattat

Answer 1

I have pushed a github repo that contains the same configuration as you mentionned in your question :

• standard android testing + emma code coverage
• robolectric + cobertura code coverage (btw thx, I thought cobertura was not compatible with robolectric)

and their inclusion in sonar. Only one at a time, but still the configuration is an interesting base to work on.

https://github.com/stephanenicolas/Quality-Tools-for-Android

A colleague and I have in mind to work on this issue with a very similar mindset as you do. We want to get better integration of different testing technologies to be available for Android projects and reporting to work in sonar as well.

If you want to join our effort, we would be happy to give you the rights to the repo and start working with you.

Stéphane

-- Update

Since march 2013, we also propose a unified configuration to use Jacoco offline instrumentation for both robolectric and standard junit tests and get results displayed simultanously in Sonar.

Lemoine

btw the post on sonar forum is mine, evgeny from sonar source is interested to tackle the issue of a better Android integration as well. – Lemoine

This is at least a start and should help others, many thanks for that! I'm pretty interested to see if Google wants to change anything with respect to testing with the new Gradle-based build... there Roadmap still lists Emma as CC tool: tools.android.com/tech-docs/new-build-system/roadmap – Rattat

This discussion is about Jacoco support for Android could change the game : github.com/jacoco/jacoco/pull/64#issuecomment-12150910. Would you like to contribute to our project ? (Or at least go on talking about it inside the issue pages of our github repo ?) – Lemoine

Sure, that would be cool. I just don't guarantee that I have time to contribute much. I'm in a crunching phase right now... my login at gh is tommyd3mdi. – Rattat

you're in. Don't worry I am in a rush too. But if you have any idea that would make the project better, please go ahead. Hope to see ya on gh. – Lemoine

Answer 2

You can use the robolectric framework. It will allow you to run JUnit Tests on the Java VM instead of using Android Junit Tests that run on the Dalvik VM. This will allow you to use any standard coverage Tool out there (EclEmma, CodeCover, Cobertura, etc.). It's easy to configure and runs stable (What you can't say from the Clover tool).

Fugazy

If you've read my initial post properly, you'd have seen that I already have Robolectric-based and covered JUnit tests in place... – Rattat

Answer 3

Disclaimer: I'm an Atlassian

Yes. There is an alternative - you can use the Atlassian Clover tool. There is an alpha version of Clover-for-Android available from https://confluence.atlassian.com/display/CLOVER/Clover-for-Android

You wrote that you want to achieve the following:

• code coverage for unit tests - yes, Clover supports "Run as > Android JUnit Test" as well as "Run as > JUnit Test" - i.e. without starting an emulator (but the latter would require slightly different setup as described in the Clover-for-Android manual)
• code coverage for integration tests - as above, "Run as > Android JUnit Test"
• ideally merging code coverage results - yes, you can merge code coverage results from multiple test runs; what's more you can even run your application in emulator or device and perform manual tests (sic!)
• publishing to Sonar - there is a Clover Sonar Plugin available, however I didn't test it with Clover-for-Android; probably you would have to run your tests in Eclipse, next export XML or HTML report (button in Clover's Coverage Explorer view) and push these files to Sonar build

Dialectal

Ah, this is all fine, I'll try it out later on. Although you should really make clear in your posts that you're working for Atlassian... – Rattat