Flutter: avoid UI freeze when massive Database operation is in progress

Question

UPDATE (15 july 2020)

mFeinstein's response, for now, is the only answer which gives me the first acceptable solution.

---

QUESTION

I have to ask you what is the best approach for doing what i'm trying to do:

1. Calling a web service in async mode
2. Parsing response
3. Performing massive database operations

All of this without freezing progress animation, like indeterminate progress bar.

There isn't problem at first and second point. Problem occurs at third, when a massive database insert is in act. And i don't understand yet how is the right way for implementing this stuff.

Some pseudo piece of code for clarify

UI (Dialog is shown and progress bar runs...)

void callWS() async {
    MyProgressDialog _dialog = DialogHelper.showMyProgressDialog(_context, "Data", "Loading...");
    await getDataFromService();
    _dialog.close();
  }

CONNECTION (Freeze doesn't occur on progress bar)

   static Future<void> getDataFromService() async {
    String uri = MY_URI;
    String wsMethod = MY_WS_METHOD;
    String wsContract = MY_WS_CONTRACT;

    SoapObject myRequest = SoapObject.fromSoapObject(namespace: my_namespace, name: wsMethod);

    MyConnectionResult response = await _openMyConnection(myRequest, uri, wsContract, wsMethod);
    if (response.result == MyResultEnum.OK) {
      await _parseResponse(response.data);
    }
  }

DATABASE (Freeze occurs on progress bar)

  static Future<void> _parseResponse(xml.XmlElement elements) async {
    Database db = await MyDatabaseHelper.openConnection();
    db.transaction((tx) async {
      Batch batch = tx.batch();
      for (xml.XmlElement oi in elements.children) {
        int id = int.parse(oi.findElements("ID").first.text);
        String name = oi.findElements("NAME").first.text;

        DatabaseHelper.insertElement(
          tx,
          id: id,
          name: name,
        );
      }
      batch.commit(noResult: true);
    });
  }

NOT WORKING ALTERNATIVE

I saw "compute" function approach too, but it seems there is a problem in sqflite package, when i call a db operation. For example:

  static Future<void> performDelete() async {
    Database db = await openMyConnection();
    compute(_performDeleteCompute, db);
  }

  static void _performDeleteCompute(Database db) async {
    db.rawQuery("DELETE MYTABLE");
  }

Console error:'
-> Unhandled Exception: Exception: ServicesBinding.defaultBinaryMessenger was accessed before the binding was initialized. 
-> If you are running an application and need to access the binary messenger before runApp() has been called (for example, during plugin initialization),
then you need to explicitly call the WidgetsFlutterBinding.ensureInitialized() first.
-> error defaultBinaryMessenger.<anonymous closure> (package:flutter/src/services/binary_messenger.dart:76:7)
    #1      defaultBinaryMessenger (package:flutter/src/services/binary_messenger.dart:89:4)
    #2      MethodChannel.binaryMessenger (package:flutter/src/services/platform_channel.dart:140:62)
    #3      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:146:35)
    #4      MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:329:12)
    #5      invokeMethod (package:sqflite/src/sqflite_impl.dart:17:13)
    #6      SqfliteDatabaseFactoryImpl.invokeMethod (package:sqflite/src/factory_impl.dart:31:7)
    #7      SqfliteDatabaseMixin.invokeMethod (package:sqflite_common/src/database_mixin.dart:287:15)
    #8      SqfliteDatabaseMixin.safeInvokeMethod.<anonymous closure> (package:sqflite_common/src/database_mixin.dart:208:43)
    #9      wrapDatabaseException (package:sqflite/src/exception_impl.dart:7:32)
    #10     SqfliteDatabaseFactoryImpl.wrapDatabaseException (package:sqflite/src/factory_impl.dart:27:7)
    #11     SqfliteDatabaseMixin.safeInvokeMethod (package:sqflite_common/src/database_mixin.dart:208:15)
    #12     SqfliteDatabaseMixin.txnRawQuery.<anonymous closure> (package:sqflite_common/src/database_mixin.dart:394:36)
    #13     SqfliteDatabaseMixin.txnSynchronized.<anonymous closure> (package:sqflite_common/src/database_mixin.dart:327:22)
    #14     BasicLock.synchronized (package:synchronized/src/basic_lock.dart:32:26)
    #15     SqfliteDatabaseMixin.txnSynchronized (package:sqflite_common/src/database_mixin.dart:323:33)
    #16     SqfliteDatabaseMixin.txnRawQuery (package:sqflite_common/src/database_mixin.dart:393:12)
    #17     SqfliteDatabaseExecutorMixin._rawQuery (package:sqflite_common/src/database_mixin.dart:126:15)
    #18     SqfliteDatabaseExecutorMixin.rawQuery (package:sqflite_common/src/database_mixin.dart:120:12)
    #19     DatabaseHelper._performDeleteCompute(package:flutter_infocad/Database/DatabaseHelper.dart:368:8)'

And also explicitly calling the WidgetsFlutterBinding.ensureInitialized() as first in runApp(), as suggested in the error log, nothing happens.

Samy

have you tried chaining the calls without await ? like db.openConnection().then() Please try all the way from the WS call – Intratelluric

I will give a try but, "then" implements a callback, "await" is a sort of "wait the end and then go on". In the first case, progress bar reaches the end quickly, it doesn't wait. In the second case, all the operations are executed in a serial way, waiting, for each one, the return value. Imagine you have 10 operations. Start by showing progress bar. Then wait for 10 conclusions. End with hiding progress bar. In second case, instead, you have to show progress bar, start 10 operations via callback, in parallel. Track for each one callback result, and when all operations return, hide progress bar – Samy

However, i think problem is the massive operation that runs on the main thread. This is why i tried to use "compute" function, without success. – Samy

please check thiis doc medium.com/flutter-community/… it says await may block the main thread. it is better to impliment the callback – Intratelluric

also this might help blog.usejournal.com/… – Intratelluric

I tried calling 2 database writing functions (with 100 INSERT operations for each one) with callback (by ".then((_){ })"). There is a problem. Callback implies 2 functions start in parallel. So there are concurrent writes on DB which imply heavy operations on CPU, so UI freeze in any case. – Samy

"async is concurrency on the same thread", second article says :(. The only good way, i guess now, is "Isolate". But sqflite package doesn't work when you move db operation inside Isolate! Damn! – Samy

Answer 1

The problem is that Flutter is Single Threaded, so once you get a heavy process running, your Single Thread will block anything else.

The solution is to be smart on how to use that Single Thread.

Dart will have an Event Queue with a bunch of Futures waiting to be processed. Once the Dart engine sees an await it will let another Future grab hold of the Single Thread and let it run. This way, one Future will run at a time inside an Isolate.

So if we get smart about it, we let everyone play at it's own time, in other words, we break down our tasks into smaller tasks, so the Dart engine won't starve other Futures, and all the processes awaiting for running can have their time.

The equivalent for your code would be something like this (assuming the for is what takes lots of time to execute, because of a large collection, and not it's individual steps):

static Future<void> _parseResponse(xml.XmlElement elements) async {
  Database db = await MyDatabaseHelper.openConnection();
  db.transaction((tx) async {
    Batch batch = tx.batch();
    for (xml.XmlElement oi in elements.children) {      
      await Future(() {
        int id = int.parse(oi.findElements("ID").first.text);
        String name = oi.findElements("NAME").first.text;

         DatabaseHelper.insertElement(
          tx,
          id: id,
          name: name,
         );
      );
    }

    batch.commit(noResult: true);
  });
}

This will fragment each step of your for loop into a Future, so at each step your UI will have the opportunity to execute whatever it needs to execute to keep your animations smooth. Keep in mind though that this will have the side effect of slowing down _parseResponse as putting each for step into the Future Event Queue will have an additional cost, so you might want to optimize this further for your particular use case.

Bosomed

flutter is so discusting. – Hemorrhoidectomy

Answer 2

Isolate & compute sometimes not working with 3rd party library, you need to use flutter_isolate

FlutterIsolate allows creation of an Isolate in flutter that is able to use flutter plugins

Ningpo

not a reliable solution, "this plugin has not been tested with a large range of plugins, only a small subset I have been using such as flutter_notification, flutter_blue and flutter_startup." – Finzer

Answer 3

this task is suited for native ios and android code, there you have real multithreading. it shouldn't take you long to implement parsing and insertion.

Finzer