Flutter integration tests for iOS on Firebase?

Question

Is it possible to run Flutter integration tests on Firebase? There seems to be conflicting info on this - some sources say its possible, but on the documentation, the iOS section seems to only be for testing on your own device.

Firebase asks for an XCT test package to be uploaded. How can it be obtained / created for a Flutter project?

Elflock

i dont understand what you want exactly ? – Treadmill

@JanviPate > Firebase asks for an XCT test package to be uploaded. How can it be obtained / created for a Flutter project? – Elflock

Answer 1

Yes, you can write Flutter integration tests for iOS on Firebase Test Lab. Even with Dart code and not Swift code. You need to use the integration_test package. You will find in the Flutter documentation how to set up the integration_test package: https://flutter.dev/docs/testing/integration-tests

Your test will look like something like this:

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import '../lib/main.dart' as app;

// How to run?
// flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  // Currently, we can't setup intergration tests, because there is a issue with integraiton test package +
  // localization.
  //
  // See this ticket for information: https://github.com/flutter/flutter/issues/84053
  testWidgets('full app test', (tester) async {
    app.main();
    await tester.pumpAndSettle();
    await tester.pump();

    // Verify that our counter starts at 0.
    expect(find.text('0'), findsOneWidget);
    expect(find.text('1'), findsNothing);

    // Tap the '+' icon and trigger a frame.
    await tester.tap(find.byIcon(Icons.add));
    await tester.pump();

    // Verify that our counter has incremented.
    expect(find.text('0'), findsNothing);
    expect(find.text('1'), findsOneWidget);
  });
}

To be able to run this test as a XCTest (this is what Firebase Test Lab needs), you need to some extra steps:

Open ios/Runner.xcworkspace in Xcode. Create a test target if you do not already have one via File > New > Target... and select Unit Testing Bundle. Change the Product Name to RunnerTests. Make sure Target to be Tested is set to Runner and language is set to Objective-C. Select Finish. Make sure that the iOS Deployment Target of RunnerTests within the Build Settings section is the same as Runner.

Add the new test target to ios/Podfile by embedding in the existing Runner target.

target 'Runner' do
  # Do not change existing lines.
  ...

  target 'RunnerTests' do
    inherit! :search_paths
  end
end

To build integration_test/foo_test.dart from the command line, run:

flutter build ios --config-only integration_test/foo_test.dart

In Xcode, add a test file called RunnerTests.m (or any name of your choice) to the new target and replace the file:

@import XCTest;
@import integration_test;

INTEGRATION_TEST_IOS_RUNNER(RunnerTests)

Run Product > Test to run the integration tests on your selected device.

To deploy it to Firebase Test Lab you can follow these steps:

Execute this script at the root of your Flutter app:

output="../build/ios_integ"
product="build/ios_integ/Build/Products"
dev_target="14.3"

# Pass --simulator if building for the simulator.
flutter build ios integration_test/foo_test.dart --release

pushd ios
xcodebuild -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath $output -sdk iphoneos build-for-testing
popd

pushd $product
zip -r "ios_tests.zip" "Release-iphoneos" "Runner_iphoneos$dev_target-arm64.xctestrun"
popd

You can verify locally that your tests are successful by running the following command:

xcodebuild test-without-building -xctestrun "build/ios_integ/Build/Products/Runner_iphoneos14.3-arm64.xctestrun" -destination id=<YOUR_DEVICE_ID>

Once everything is ok, you can upload the resulting zip to Firebase Test Lab (change the model with your values):

gcloud firebase test ios run --test "build/ios_integ/ios_tests.zip" --device model=iphone11pro,version=14.1,locale=fr_FR,orientation=portrait

Note: I copied to the needed extra for iOS device tests steps from https://github.com/flutter/flutter/tree/master/packages/integration_test#ios-device-testing (credits to the Flutter team ❤️)

If you want to run these tests on Android Firebase Test Lab, you need also to do some extra steps. You will find these steps under: https://github.com/flutter/flutter/tree/master/packages/integration_test#android-device-testing. I can also recommend to check out this video, where Reso Coder explained and showed every little step: https://www.youtube.com/watch?v=izajHHFSa8o

Beaux

Execution of test on firebase test lab fails with error: The XCTest zip file was malformed. The zip did not contain a single .xctestrun file and the contents of the DerivedData/Build/Products directory.. – Capsulize

Answer 2

First of all open your ios Module in Xcode as ss below.

Now you are free to write cool XCode UI tests with Swift. For example I am dropping here a Firebase Login test for your app maybe you wish to use.

import XCTest
import UIKit
import Firebase

class LoginTestUITests: XCTestCase {

    var app: XCUIApplication!
    
    func testLogin() {
        continueAfterFailure = false
        app = XCUIApplication()
        app.launch()
        wait(for: 2)
        passLogin()
        wait(for: 5)
     // after your login you can add some methods here according to your app flow
    }
}

extension LoginTestUITests {

func passLogin() {
    
    //put your logic here        
    
}
// this is a pretty cool extension waiting according to input for a while coming the data from backend etc.
extension XCTestCase {
    func wait(for duration: TimeInterval) {
        let waitExpectation = expectation(description: "Waiting")
        
        let when = DispatchTime.now() + duration
        DispatchQueue.main.asyncAfter(deadline: when) {
            waitExpectation.fulfill()
        }
        // I use a buffer here to avoid flakiness with Timer on CI
        waitForExpectations(timeout: duration + 0.5)
    }
}

Evolute

Also there is a nice sample case which explain how to write for swiping test on different screens. swiftbysundell.com/articles/… – Evolute

So there's no way to write the tests in Dart / Flutter and have those be used? I'm guessing these swift tests can't be used to interact with buttons / UI elements created using Flutter – Elflock

I think you need to implement Swifty codes. – Evolute

Answer 3

Open Xcode project (by default, it's ios/Runner.xcodeproj). Create a test target (navigating File > New > Target... and set up the values) and a test file RunnerTests.m and change the code. You can change RunnerTests.m to the name of your choice.

#import <XCTest/XCTest.h>
#import <integration_test/IntegrationTestIosTest.h>

INTEGRATION_TEST_IOS_RUNNER(RunnerTests)

Treadmill

This is the same info as in the docs but this doesn't explain: 1) Will this run my flutter integration tests? 2) How do I obtain the XCT package that Firebase asks for? – Elflock

Answer 4

You need to have Xcode to do it. If you are using Android first open iOS module in Xcode

Then in Xcode open the Test navigator (Command-6) and add new test target:

Now we have two targets for Unit and UI test in our Flutter project

Our project is ready for test and we can check it run Product ▸ Test or Command-U in Xcode:

Then Firebase Test Lab:

• iOS: Write XCTests, then build and package your app for upload.

Just create application package using Xcode Product > Archive selecting real device as a target.

Stinger

Do I have to write the tests in Swift? Or is there a way to write my tests in Dart using flutter code, and have those hooked to XCTests somehow which Firebase can run? – Elflock

Looks like need to write in Swift, did not find anything for Dart but it should not be very difficult. There is the link that should help raywenderlich.com/… , cheers – Stinger