Firebase Test Lab Coverage with Orchestrator Permission Denied

Question

We are currently attempting to use Google's Firebase Test Labs for testing our android app on devices. To test things, I've created a small app that doesn't do anything except get tested, and it's worked well so far, except that the recommended way (taken directly from the documentation page) to achieve code coverage, does not seem to work properly with the orchestrator.

With the following gcloud command:

gcloud firebase test android run --results-bucket=<hidden> --use-orchestrator --type instrumentation --device model=Pixel2,version=29 --device model=Pixel2,version=28 --app app/build/outputs/apk/debug/app-debug.apk  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --environment-variables clearPackageData=true,coverage=true,coverageFilePath=/sdcard/  --directories-to-pull=/sdcard/

the tests run successfully, but for every single test execution, the creation of the coverage execution data fails. The logcat entry looks like:

04-08 09:50:37.874: E/CoverageListener(8114): Failed to generate Emma/JaCoCo coverage. 
04-08 09:50:37.874: E/CoverageListener(8114): java.lang.reflect.InvocationTargetException
04-08 09:50:37.874: E/CoverageListener(8114):   at java.lang.reflect.Method.invoke(Native Method)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
04-08 09:50:37.874: E/CoverageListener(8114):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
04-08 09:50:37.874: E/CoverageListener(8114): Caused by: java.io.FileNotFoundException: /sdcard/com.example.citest.AnotherInstrumentedTest#testSdkVersion.ec (Permission denied)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.open0(Native Method)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.open(FileOutputStream.java:308)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.<init>(FileOutputStream.java:238)
04-08 09:50:37.874: E/CoverageListener(8114):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
04-08 09:50:37.874: E/CoverageListener(8114):   ... 7 more

This indicated to me that the app does not have permission to write to external storage, but it still does not work after adding

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>

to both the app's manifest, as well as the additional manifest in the androidTest folder.

When changing the command to no longer use the orchestrator, and instead use the single coverage file as indicated in the documentation:

gcloud firebase test android run --results-bucket=<hidden> --no-use-orchestrator --type instrumentation --device model=Pixel2,version=29 --device model=Pixel2,version=28 --app app/build/outputs/apk/debug/app-debug.apk  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --environment-variables coverage=true,coverageFile=/sdcard/coverage.ec  --directories-to-pull=/sdcard/

The coverage file for the device using version 28 is successfully created, but for the device using version 29, it still fails with

04-08 05:07:43.341: E/CoverageListener(9398): Failed to generate Emma/JaCoCo coverage. 
04-08 05:07:43.341: E/CoverageListener(9398): java.lang.reflect.InvocationTargetException
04-08 05:07:43.341: E/CoverageListener(9398):   at java.lang.reflect.Method.invoke(Native Method)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
04-08 05:07:43.341: E/CoverageListener(9398):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
04-08 05:07:43.341: E/CoverageListener(9398): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec: open failed: EACCES (Permission denied)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.IoBridge.open(IoBridge.java:496)
04-08 05:07:43.341: E/CoverageListener(9398):   at java.io.FileOutputStream.<init>(FileOutputStream.java:235)
04-08 05:07:43.341: E/CoverageListener(9398):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
04-08 05:07:43.341: E/CoverageListener(9398):   ... 7 more
04-08 05:07:43.341: E/CoverageListener(9398): Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.Linux.open(Native Method)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
04-08 05:07:43.341: E/CoverageListener(9398):   at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7255)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.IoBridge.open(IoBridge.java:482)
04-08 05:07:43.341: E/CoverageListener(9398):   ... 9 more

In all cases, simply running the android device tests on a local emulator via gradle cAT, produces valid coverage files, and the problems only occur when running the tests via the Firebase Test labs.

Gundry

Did you find the solution? – Ambo

I've same problem with Orchestrator+set permissions, but still error about lack of perms – Ambo

Answer 1

The coverage file for the device using version 28 is successfully created, but for the device using version 29, it still fails with

So, I was running into this same issue, and it was this comment that tipped me off to the solution.

Basically in Android 10+ they have changed the way that apps interact with external devices to a "scoped" approach. More details here: https://developer.android.com/training/data-storage

As far as I can tell whichever library is trying to write the coverage.ec to the sdcard, is not doing it correctly as per this "scoped" approach.

So by adding this to your AndroidManifest.xml:

<application
  android:requestLegacyExternalStorage="true">
</application>

It will tell Android to use the old non-"scoped" method of writing to the SD card, and then I found this to work correctly.

I think this is only temporary though, and this might not work with Android 11. But some library needs to update to change how it writes to the /sdcard.

Temporal

This works partially: Without the orchestrator, I can now run the tests. However, when I enable the orchestrator, I still get permission denied errors on both Android 28 and 29. Are you using the orchestrator, or is that disabled in your case? – Gundry

@Gundry I don't use the orchestrator, so yes it's disabled in my case, not sure what the difference there would be – Temporal

The orchestrator basically runs each Test class on a clean slate, and instead of one single coverage file, one coverage file per test class is generated. However, these files still fail to be written with a permission denied error, on both 28 and 29, regardless of requestLegacyExternalStorage – Gundry

@Gundry did you set the path for orchestrator to save the files? (using coverageFilePath) – Tramroad

Answer 2

I've answered the same question here:

https://mcmap.net/q/816308/-how-to-get-code-coverage-reports-from-google-firebase-for-android-espresso-tests

tl;dr

Add

@get:Rule
val runtimePermissionRule: GrantPermissionRule = GrantPermissionRule.grant(
    Manifest.permission.WRITE_EXTERNAL_STORAGE,     
    Manifest.permission.READ_EXTERNAL_STORAGE
)

Ambo

Did this really solve your issue with orchestrator? What android versions? – Coldiron

It did, it just has to be in correct order (before) with the ActivityScenarioRule – Coldiron