Running multiple test case on Firebase Test Lab

Question

I am trying to run my integration test on Firebase Test Lab.

flutter build apk -t lib/main_dev.dart
./gradlew app:assembleAndroidTest -Ptarget=lib/main_dev.dart
./gradlew app:assembleDebug -Ptarget=integration_test/login_test.dart

This code generates app-debug-androidTest.apk and app-debug.apk and once I upload them in Test Lab Test are executed perfectly.

Now the issue is I have many test files under integration_test. I am not sure how to create a app-debug-androidTest.apk that includes all the testcases under integration_test.

I did try the following:

flutter build apk -t lib/main_dev.dart
./gradlew app:assembleAndroidTest -Ptarget=lib/main_dev.dart
./gradlew app:assembleDebug -Ptarget=test_driver/integration_test.dart

but this stuck test at black screen which is I thing weird but a correct behavior as while running the integration test in local device also we need to provide target along with the driver.

So for Local I have a script

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/login_test.dart

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/register_test.dart

which runs the all the Integration code.

So my question is how to upload all the test cases.

Or do we have to make build for each test case

./gradlew app:assembleDebug -Ptarget=integration_test/login_test.dart

then upload it to the Test Lab then again

./gradlew app:assembleDebug -Ptarget=integration_test/register_test.dart

and upload again?

Weighting

Answer 1

I have just encountered the same issue. For future reference for other people, what I did was to import all the tests into one single file all_tests.dart.

Let's say the directory structure is

integration_test/
  test1.dart
  test2.dart
  test3.dart
  all_tests.dart

In all_tests.dart, I imported all the tests in:

import 'test1.dart' as test1;
import 'test2.dart' as test2;

void main() {
  test1.main();
  test2.main();
  ...
}

Hubblebubble

Is this working? I get to launch only the first file, then I get "All tests passed." and the other are skipped. – Microbe

EDIT: It actually works, but not if you're using the screenshot approach explained here: github.com/flutter/flutter/tree/master/packages/… – Microbe

It's not working for me, it is said "No tests found". Could you share a more detailed gist? Btw it's only working on flutter integration test (flutter drive) – Lunation

With the new flutter integration test, you can run the whole folder I think. Just need to specify the directory, that's it. @Lunation – Hubblebubble

@AnhPham it is indeed working if I run it from Vscode, all of the integration tests are running. But it is not working if I used the instrumentation test as the OP specified – Lunation

This approach didn't work for us. Even when running couple of test files together. While running the big suite with flutter test integration_test/ we observed that between test run there is an additional build happening, while slow that step is creating reliability – Runyon

Appending the above comment, I was able to get rid of flakiness by creating additional groups within tests. We had setup/teardown with getit DI setup and that was getting messed up without group. – Runyon

Answer 2

I had the same kind of problem where there was no real way to get back to a clean app state after a test was run. Clearing the app data between the tests helped me a lot.

With

import 'dart:io';
import 'package:path_provider/path_provider.dart';

tearDown(() async {
  final filesDirPath = (await getApplicationSupportDirectory()).path;
  if (Directory(filesDirPath).existsSync()) {
    await Directory(filesDirPath).delete(recursive: true);
  }
});

There could be more directories to remove like getTemporaryDirectory(), but this is usually where the user data is saved.

Capriccioso