Run Flutter Package unit tests serially one by one

Question

I'm writing a bunch of unit tests with an HTTP client in them, for a custom Flutter package.

I noticed that when I run the tests with flutter test, the first two unit tests will start at approximately the same time.

This is not something I want. Because the unit tests are supposed to write some data, and at the start of every unit test the data is reset. That way every test starts off with the same data.

But since there are two tests running at the same time, they both access the same file and corrupt it or not get access to it with FileSystemException: lock failed.

Is there any way to force the tests to run one by one, instead of multiple at once? I tried putting them in separate files, but that did not work.

Thanks

Kenton

are you are using groups in your test? – Tolmach

Answer 1

If you have several tests that are related to one another, combine them using the group function provided by the test package.

Please check https://flutter.dev/docs/cookbook/testing/unit/introduction#5-combine-multiple-tests-in-a-group

Tolmach

Putting tests in a group did not cause them to run sequentially for me. I also moved setUp and tearDown in the group with no change. – Cornemuse

Answer 2

By default, the flutter test command executes the tests concurrently, but you can specify the concurrency using -j, --concurrency=<jobs> in the flutter test command.

As per the Flutter help document:

-j, --concurrency=<jobs> defines the number of concurrent test processes to run. This will be ignored when running integration tests. (defaults to "14")

execute the below command to run all tests one by one

flutter test -j, --concurrency=1

execute the below command to run all tests one by one with coverage,

flutter test --coverage -j, --concurrency=1

Citric

Answer 3

I ran into a similar issue where I needed a bunch of tests to run in sequence. I put together this little thing approximating semaphores using futures which worked for my use case, maybe it's useful for someone else too

You create a "sequence", and then you wrap the tests in runInSequence(...) like this

void main() {
  final seq = newTestSequence();

  test('test 1', () async {
    await runInSequence(seq, () async {
      print('starting test 1...');
      await Future.delayed(const Duration(milliseconds: 1000));
      print('test 1 done');
    });
  });

  test('test 2', () async {
    await runInSequence(seq, () async {
      print('starting test 2...');
      await Future.delayed(const Duration(milliseconds: 10));
      print('test 2 done');
    });
  });
}

This is the code for newSequence and runInSequence

import 'dart:async';
import 'package:flutter/widgets.dart';

final Map<Key, Completer> _semaphores = {};

Key newTestSequence() {
  final sequence = GlobalKey();

  // create a new "semaphore" that will allow the first awaiter through
  // immediately
  _semaphores[sequence] = Completer();
  _semaphores[sequence]!.complete();

  return sequence;
}

Future<void> runInSequence(Key sequence, Future Function() test) async {
  // block until resource is acquired
  final c = await _claimSemaphore(sequence);

  // execute test
  await test();

  // release resource
  c.complete();
}

Future<Completer> _claimSemaphore(Key sequence) async {
  if (!_semaphores.containsKey(sequence)) {
    throw "unknown sequence $sequence; create one using `newTestSequence`";
  }

  final Completer c = _semaphores[sequence]!;

  // block execution until the future is completed
  await c.future;

  // replace completer because futures can only be completed once
  _semaphores[sequence] = Completer();

  return _semaphores[sequence]!;
}

Kremenchug