Inserting large quantities in IndexedDB's objectstore blocks UI

Question

I want to save some ~35000 objects in my IndexedDB's objectstore. I am using below code to insert.

AddListings = function (x2j_list_new, callback) {   
    var transaction = db.transaction(["listings"], IDBTransaction.READ_WRITE);
    var count = 0;
    transaction.oncomplete = function (event) {
        if (callback) {
            console.log('x2jShowListing Added ' + count + '/' + x2j_list_new.length);
                callback([count, x2j_list_new.length]);
            }
    };
    transaction.onerror = function (e) {
       console.log("myError: ", e);  
       if (callback) {
          callback(false);
       }
    };
    var store = transaction.objectStore("listings");

    $.each(x2j_list_new, function (index0, item0) {
        var request = store.put(item0);
        request.onsuccess = function (event) {
            count++;
            // event.target.result  
            };
        });
    });        
};

The above code works fine, but looping and inserting over ~35000 objects makes the UI unresponsive for ~200 seconds. I thought maybe i can use WebWorkers, but IndexedDB is not available inside WebWorkers. I tried to find a way to bulk insert, couldn't find one. Any ideas of how to insert large quantities of objects without blocking the UI?

Intricacy

For now I am splitting the array in chunks of 500 and using setInterval instead of for loop. Now the UI responds little better than before. – Intricacy

So I know this is old but just wanted to know if anyone had updates on using IndexedDB in Web workers since it is supported now? – Archibaldo

Answer 1

You're on the right track, but you're asking the browser to store 35,000 objects before it's had a chance to finish storing one. Here's code which asynchronously waits for one request to finish before starting the next (but using the same transaction):

    openRequest = window.indexedDB.open("MyDatabase", 1);
    openRequest.onerror = function(event) {
        console.error(event);
    };
    openRequest.onsuccess = function (event) {
        var db = openRequest.result;
        db.onerror = function(event) {
            // Generic error handler for all errors targeted at this database's requests
            console.error(event.target);
            window.alert("Database error: " + event.target.wePutrrorMessage || event.target.error.name || event.target.error || event.target.errorCode);
        };
        var transaction = db.transaction('item', "readwrite");
        var itemStore = transaction.objectStore("item");
        putNext();

        function putNext() {
            if (i<items.length) {
                itemStore.put(items[i]).onsuccess = putNext;
                ++i;
            } else {   // complete
                console.log('populate complete');
                callback();
            }
        }           
    };      

Sticky

To be sure that all items are really in the store - subscribe to transaction.oncomplete event, smth like: transaction.oncomplete = callback – Skimmer

I can't see how to determine the end of a transaction, one get a store from transaction then add a one or more records... How do I make the transaction end after I add all of it ? Does the transaction ends after onsuccess callback returns? Which is after all entries are added in this recursive call? I suppose so – Weatherley

Answer 2

You're doing everything right by using callbacks.

The Webworker API has not yet been implemented by any major browser. Interestingly, it is expected to be synchronous. The regular API is async for the exact reason you describe -- it's not supposed to block the UI thread.

Using callbacks is the way to avoid lock ups, but at 35k objects you're clearly seeing this paradigm break down. Unfortunately, IDB performance is not yet on par with WebSQL from the benchmarks I've seen.

With Chrome's LevelDB there's been some new experimental backends (FF is SQLite ) but I think your experience proves that there's some room for improvement.

Specht

Answer 3

Wild guess from my side, but if WebSQL is available from what is known as a "background page", and assuming the bandwidth of messaging between the front- and backpage does not lock up the UI in the same way, maybe a background page with intra page message could be utilized?

Mylohyoid

I have a feeling that its the for loop which is causing block UI. I wish there was a way to ping browser in regular interval so it doesnot think its stuck – Intricacy

In traditional GUI programming, you could detect when the browser is "idle" and execute another step in your big foor loop. Unfortunately, detecting the idle state is not easy in javascript, but there are hacks to simulate it. For instance #668055 . – Mylohyoid

Answer 4

I am splitting the array in chunks of 500 and using setTimeout instead of for loop. Now the UI responds little better than before

Intricacy

I don't recommend accessing IndexedDB from multiple event loop tasks. It behaves pretty weird this way – Weatherley

Answer 5

do it all in one transaction, for example

    function populateData (db, storetable, records, whencomplete, whenfails) 
    {
        var transaction = db.transaction (storetable, 'readwrite');
        var objectStore = transaction.objectStore (storetable);

        for (var ii = 0; ii < records.length ; ii ++) 
            objectStore.put (records[ii]);

        if (whencomplete)
            transaction.oncomplete = function (event) { whencomplete (event);  }

        if (whenfails)
            transaction.onerror = function (event) { whenfails (event);  }
    }
    

see also function populateData in the example https://github.com/mdn/dom-examples/blob/main/indexeddb-examples/idbcursor/scripts/main.js

Aedes