Coverage for android tests using orchestrator

Question

I am working on a project where in we are trying to use the ANDROID TEST ORCHESTRATOR for it's obvious benefits of isolating crashes. But while executing the test suite, it appears to me that as the orchestrator initiates a new process for every test case, the test coverage report for the suite execution is always showing incomplete data (mostly the data for the last test case present in the test suite).

So I was wondering that is there a way to overcome this problem and generate a jacoco code coverage report for all instrumented tests existing in the test suite.

Greathouse

I am still searching for answers to this, any idea is also invited, does not need to provide an exact solution. – Greathouse

I found an issue similar to this or the same where I needed to remove orchestration to get androidTest code coverage. Still haven't resolved it. You can see our gradle here: #48427786 We too still haven't worked out the orchestration bit and would like to know as well. Here is the only related tracker issue I could find: issuetracker.google.com/issues/71989799 – Frae

Logged this bug: issuetracker.google.com/issues/72758547 – Frae

@Greathouse What did you end up doing ? – Schizomycete

@IslamSalah only observation from my end is that for each and every test case a separate coverage file is being generated, so currently i am building a system to keep on listening to the path (where the coverage file gets dumped), and then collate all the coverage reports into one. But hope google comes out with a more optimised solution soon. Let me know if you are trying something else as well. – Greathouse

Google has made a beta release of orchestrator to fix the issue at their end (more info at developer.android.com/training/testing/release-notes), but still Gradle and Android Studio teams need to pickup and implement the fix at their end to provide a completely working solution. – Greathouse

Answer 1

If you're using the android test orchestrator this is the problem. There is an open bug report: https://issuetracker.google.com/issues/72758547

The only solution that I know of is to disable the android test orchestration until a fix is released.

In my Android Java build.gradle I had to comment out the test orchestrator like so:

android {
...
    testOptions {
        // temporarily disable the orchestrator as this breaks coverage: https://issuetracker.google.com/issues/72758547
        //execution 'ANDROID_TEST_ORCHESTRATOR'
    ...
    }
}

Psychological

Thanks for pointing out the obvious, but I am trying to find a way to overcome the bug with some workaround as the app in test at my end does not persist without crashing off while the complete test suite execution. – Greathouse

Whops - didn't realise you already knew about the bug report! Might still help someone who like me just wanted their coverage report to start working again :) – Psychological

Yeah that's true :) Thanks anyway, let me know if you come across something more about it. – Greathouse

Answer 2

This seems to be fixed with a few new params, as shown in this sample: https://github.com/android/testing-samples/tree/main/runner/AndroidTestOrchestratorWithTestCoverageSample

I just needed to bump this library: coreVersion = "1.6.0-alpha05", besides that it worked as is.

---

We need to add the orchestrator:

androidTestUtil "androidx.test:orchestrator:" + rootProject.orchestratorVersion
androidTestUtil "androidx.test.services:test-services:" + rootProject.testServicesVersion

Enable the useTestStorageService and clearPackageData flags on the runner:

testInstrumentationRunnerArguments clearPackageData: "true"
testInstrumentationRunnerArguments useTestStorageService: "true"

Setup the flags for code coverage and orchestrator:

buildTypes {
    debug {
        testCoverageEnabled true
    }
}

testOptions {
    execution "ANDROIDX_TEST_ORCHESTRATOR"
}

And lastly, make sure you're pointing to the right *.ec files, if you're using a custom JaCoCo task, this one is not shown on the GitHub sample.

include("outputs/code_coverage/debugAndroidTest/connected/*/*.ec")

---

If that's still not working, some of the libraries and versions might need to be tweaked, these are the ones that worked for me (most of them come as is on the repository):

ext {
    androidxAnnotationVersion = "1.2.0"
    guavaVersion = "30.1.1-android"
    coreVersion = "1.6.0-alpha05"
    extJUnitVersion = "1.2.0-alpha01"
    runnerVersion = "1.6.0-alpha03"
    rulesVersion = "1.6.0-alpha01"
    testServicesVersion = "1.5.0-alpha01"
    orchestratorVersion = "1.5.0-alpha01"
    espressoVersion = "3.6.0-alpha01"
    truthVersion = "1.1.3"
}

Skid

Answer 3

The issue mentioned above is now fixed and the latest gradle version is working fine for all android devices.

Greathouse