How to get code coverage reports from google Firebase for Android Espresso tests

Asked 21/11, 2016 at 17:42 Answered 4/5 at 14:27

android firebase code-coverage android-espresso jacoco

Based on this documentation - https://developer.android.com/studio/test/command-line.html#AMOptionsSyntax it is possible to get code coverage results back from Firebase lab. Some folks in #test-lab at firebase-community.slack.com are able to get it working but after few attempts I am still hitting a wall.

I am able to get a combined code coverage report of jacaco and emma by following this guide so there is nothing wrong with my local setup but problematic when trying to give arguments to gcloud cmd line to ask for coverage numbers and it talks about emma coverage.

Essentially, when I run this command locally

gcloud beta test android run \
  --type instrumentation \
  --app app/build/outputs/apk/*-debug-unaligned.apk \
  --test app/build/outputs/apk/*-debug-androidTest-unaligned.apk \
  --device-ids Nexus6\
  --os-version-ids 22 \
  --locales en \
  --orientations portrait \
  --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \
--directories-to-pull=/sdcard

I am expecting the coverage report to be generated but I get this in the instruments.results file

INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=
com.godaddy.gdm.telephony.uitests.DialerTabTest:
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: stream=

Time: 6.022

OK (1 test)


Error: **Failed to generate emma coverage.**
INSTRUMENTATION_CODE: -1

And logcat says this:

11-18 21:38:39.400: I/TestRunner(5246): run finished: 1 tests, 0 failed, 0 ignored
11-18 21:38:39.400: I/TestRunner(5246): [ 11-18 21:38:39.400  5246: 5263 E/         ]
11-18 21:38:39.400: I/TestRunner(5246): Failed to generate emma coverage.
11-18 21:38:39.400: I/TestRunner(5246): java.lang.reflect.InvocationTargetException
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Method.java:372)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:80)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:68)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:94)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:69)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
11-18 21:38:39.400: I/TestRunner(5246):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
11-18 21:38:39.400: I/TestRunner(5246): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:456)
11-18 21:38:39.400: I/TestRunner(5246):   at java.io.FileOutputStream.<init>(FileOutputStream.java:87)
11-18 21:38:39.400: I/TestRunner(5246):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
11-18 21:38:39.400: I/TestRunner(5246):   ... 8 more
11-18 21:38:39.400: I/TestRunner(5246): Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.Posix.open(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.BlockGuardOs.open(BlockGuardOs.java:186)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:442)
11-18 21:38:39.400: I/TestRunner(5246):   ... 10 more

I can give more information if needed.

Powerboat answered 21/11, 2016 at 17:42 Comment(4)

How do you get the code coverage file on your local machine? – Indubitable 2/3, 2018 at 22:43

It is usually in the app/build/outputs/<buildflavor>/jacoco/coverage.ec file. For exact one try a search for coverage.ec in your project folder – Powerboat 4/3, 2018 at 2:45

Right, but when you run it with firebase it's not going to be there on your local machine, right? I'm asking how do you get it off of firebase onto your local machine? – Indubitable 4/3, 2018 at 4:5

oh you need gsutil command. Take a look at medium.com/mobile-testing/… – Powerboat 6/3, 2018 at 21:14

It turned out all that needed was to enable the proper permission for Writing on SD card.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

in AndroidManifest.xml like in this SO answer

Also, make sure testCoverageEnabled true is enabled in app/build.gradle for debug

debug {
            testCoverageEnabled true
}

from this medium post

Powerboat answered 17/1, 2017 at 0:28 Comment(2)

Remember to add this permission just in debug build type if your app doesn't need this for other reason – Micheal 16/3, 2018 at 10:19

If you enable the orchestrator using flag --use-orchestrator, you should use "coverageFilePath" rather than "coverageFile" to get all the .ec files. Otherwise you will get an exception. check release notes of AndroidTestOrchestrator 1.0.2-beta1 And I also didn't get success for a file path other than "/sdcard/coverage.ec" – Hart 17/7, 2018 at 13:8

In case somebody meets this on Android 10, on Firebase Test Lab and above answer doesn't work, try adding grant rule (along with permission in Manifest and testCoverageEnabled true).

@get:Rule
val runtimePermissionRule: GrantPermissionRule = GrantPermissionRule.grant(
    Manifest.permission.WRITE_EXTERNAL_STORAGE,     
    Manifest.permission.READ_EXTERNAL_STORAGE
)

Kilowatt answered 27/11, 2020 at 9:19 Comment(0)

Rather than adding a new permission, what worked for me was to specify the path to the internal storage of the app, for coverageFilePath.

In my case it's configured in my app/build.gradle:

        // The following argument makes the Android Test Orchestrator run its
        // "pm clear" command after each test invocation. This command ensures
        // that the app's state is completely cleared between tests.
        // https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#use-android
        testInstrumentationRunnerArguments clearPackageData: 'true', coverage: 'true', coverageFilePath: '/data/data/com.example.myapp/'

The coverage files are then found in:

app/build/outputs/code_coverage/debugAndroidTest/connected/Pixel_4a_API_34_google_apis(AVD) - 14/com.example.myapp.MyTestClass#myTestClass.ec

UPDATE: In fact, only the coverage file of the last test class seems to be retrieved when clearPackageData is set. If clearPackageData isn't set, then all coverage files are retrieved.

Samaria answered 23/9, 2023 at 22:56 Comment(0)

As of April 2024, the Google documentation for Firebase Test Lab recommends using the file path coverageFile="/sdcard/Download/coverage.ec" when starting a UI Instrumentation Test. This is the path where the code coverage data will be stored on the device, when the test suite finishes.

Documentation: https://firebase.google.com/docs/test-lab/android/command-line#code_coverage_reports_for_instrumentation_tests

gcloud firebase test android run \
    --type instrumentation \
    --app "./app/build/outputs/apk/debug/app-debug.apk" \
    --test "./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk" \
    --device "model=F01L,version=27,locale=en_US" \
    --environment-variables coverage=true,coverageFile="/sdcard/Download/coverage.ec" \
    --directories-to-pull "/sdcard/Download" \
    --timeout 20m

When the test finishes, the files are uploaded to Google Cloud Storage, and you must use the gsutil command to retrieve the coverage.ec file from Google Cloud Storage, to generate a code coverage report.

Unfortunately, the folder path "/sdcard/Download/" is difficult to handle on Android 6, 7, 8, 9, and 10, because it's a protected area with high security. And the Google documentation does not say how to handle it. If you don't handle it, the test app will not have permission to write the file, and there will be no "artifacts" folder and no coverage.ec file stored on the Firebase device. So you will see the following errors when the test suite finishes:

gsutil - error

# Get a URL to the current running Firebase test (the most recent one, which is running now)
CURRENT_TEST_URL=$(gsutil ls "gs://test-lab-YOUR-TEST-LAB-PROJECT-PATH/" | tail -1)
# List all the code coverage binary files from Firebase Cloud Storage ("**" means from any folder).
gsutil ls "${CURRENT_TEST_URL}**/*coverage*"
Error: CommandException: One or more URLs matched no objects.

Logcat log - search for "coverage"

E InstrumentationCoverageReporter: Failed to generate Emma/JaCoCo coverage.
E InstrumentationCoverageReporter: java.lang.reflect.InvocationTargetException
E InstrumentationCoverageReporter: at java.lang.reflect.Method.invoke(Native Method)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.coverage.InstrumentationCoverageReporter.generateCoverageInternal(InstrumentationCoverageReporter.java:218)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.coverage.InstrumentationCoverageReporter.dumpCoverageToFile(InstrumentationCoverageReporter.java:114)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.coverage.InstrumentationCoverageReporter.generateCoverageReport(InstrumentationCoverageReporter.java:92)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:91)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:99)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:72)
E InstrumentationCoverageReporter: at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
E InstrumentationCoverageReporter: at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:446)
E InstrumentationCoverageReporter: at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2209)
E InstrumentationCoverageReporter: Caused by: java.io.FileNotFoundException: /sdcard/Download/coverage.ec: open failed: EACCES (Permission denied)
E InstrumentationCoverageReporter: at libcore.io.IoBridge.open(IoBridge.java:496)
E InstrumentationCoverageReporter: at java.io.FileOutputStream.<init>(FileOutputStream.java:235)
E InstrumentationCoverageReporter: at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
E InstrumentationCoverageReporter: ... 10 more
E InstrumentationCoverageReporter: Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
E InstrumentationCoverageReporter: at libcore.io.Linux.open(Native Method)
E InstrumentationCoverageReporter: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
E InstrumentationCoverageReporter: at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)
E InstrumentationCoverageReporter: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
E InstrumentationCoverageReporter: at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7893)
E InstrumentationCoverageReporter: at libcore.io.IoBridge.open(IoBridge.java:482)

instrumentation.results file - at the end of the file

INSTRUMENTATION_RESULT: coverageFilePath=/sdcard/Download/coverage.ec
INSTRUMENTATION_RESULT: stream=

Time: 704.635
OK (277 tests)

Error: Failed to generate Emma/JaCoCo coverage.

Here are the steps to handle the folder path `"/sdcard/Download/"` to allow write access:

For Android 6, 7, 8, 9 (API 23-28)

Add this to your app/src/debug/AndroidManifest.xml:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" android:maxSdkVersion="29"/>

For Android 10 (API 29)

For some reason, Android 10 needs an additional line. Add these to your app/src/debug/AndroidManifest.xml:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" android:maxSdkVersion="29"/>
<application android:requestLegacyExternalStorage="true" />

You can test this permission locally on an Android emulator or physical device, by trying to create a file "/sdcard/Download/test.txt" in a UI test.

@Test
public void testThatSdCardDownloadIsAccessible() throws IOException {
    File testFile = new File("/sdcard/Download/test.txt");
    boolean result = testFile.delete();
    boolean createFileResult = testFile.createNewFile();
    assertTrue(testFile.exists());
}

For Android 11, 12, 13, 14+

No changes or permissions are needed because on these Android versions, the folder path "/sdcard/Download/" is accessible to everyone.

Sources and more info:

https://proandroiddev.com/android-code-coverage-on-firebase-test-lab-part-1-the-basics-9e1492ec5399 -- "Android Code Coverage on Firebase Test Lab (Part 1: The Basics)" by Aidan Low, 28 November 2021
https://github.com/android/testing-samples/issues/165#issuecomment-983272102 -- "Firebase Test Lab example that demos Jacoco support #165"

Causation answered 4/5 at 14:27 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++