Synchronizing MongoDB server data to an IndexedDB local store

Question

I'm trying to evaluate using IndexedDB to solve the offline issue. It would be populated with data currently stored in a MongoDB database (as is).

Once data is stored in IndexedDB, it may be changed on the MongoDB server and I need to propagate those changes. Is there any existing framework or Library to do somehting like this for Mongo. I already know about CouchDB/PouchDB and am not exploring those two.

Maryrose

You should see if any of the code from meteor is relevant. – Underdog

Very interesting, thanks. Possibly a solution to my real problem : writing an offline app. You should make this an answer so I can give you the points if it does solve my problem – Maryrose

Turns out the complete offline scenario isn't supported (offline login + changes + sync to server when back online) : groups.google.com/d/topic/meteor-core/ZVj6RFbKDq0/discussion – Maryrose

So this app will never run online? – Underdog

Yes it will but it must be possible to login & do stuff offline. Essentially downloding a part of the database locally and pushing change requests to the server when it connects. – Maryrose

Since it's now 2016 and your question has only become more relevant, have you found any solution or is it only CouchDB/PouchDB that currently supports this? – Quintin

Answer 1

[Sync solution for 2021]

I know the question asked was for MongoDB specifically, but since this is an old thread I thought readers might be looking for other solutions for new apps or rebuilds. I can really recommend to check out AceBase because it does exactly what you were looking for back then.

AceBase is a free and open source realtime database that enables easy storage and synchronization between browser and server databases. It uses IndexedDB in the browser, its own binary db / SQL Server / SQLite storage on the server. Offline edits are synced upon reconnect and clients are notified of remote database changes in realtime through a websocket (FAST!).

On top of this, AceBase has a unique feature called "live data proxies" that allow you to have all changes to in-memory objects to be persisted and synced to local and server databases, and remote changes to automatically update your in-memory objects. This means you can forget about database coding altogether, and code as if you're only using local objects. No matter whether you're online or offline.

The following example shows how to create a local IndexedDB database in the browser, how to connect to a remote database server that syncs with the local database, and how to create a live data proxy that eliminates further database coding. AceBase supports authentication and authorization as well, but I left it out for simplicity.

const { AceBaseClient } = require('acebase-client');
const { AceBase } = require('acebase');

// Create local database with IndexedDB storage:
const cacheDb = AceBase.WithIndexedDB('mydb-local');

// Connect to server database, use local db for offline storage:
const db = new AceBaseClient({ dbname: 'mydb', host: 'db.myproject.com', port: 443, https: true, cache: { db: cacheDb } });

// Wait for remote database to be connected, or ready to use when offline:
db.ready(async () => {

    // Create live data proxy for a chat:
    const emptyChat = { title: 'New chat', messages: {} };
    const proxy = await db.ref('chats/chatid1').proxy(emptyChat);  // Use emptyChat if chat node doesn't exist

    // Get object reference containing live data:
    const chat = proxy.value;

    // Update chat's properties to save to local database, 
    // sync to server AND all other clients monitoring this chat in realtime:
    chat.title = `Changing the title`;
    chat.messages.push({ 
        from: 'ewout', 
        sent: new Date(),
        text: `Sending a message that is stored in the database and synced automatically was never this easy!` +
              `This message might have been sent while we were offline. Who knows!`
    });

    // To monitor and handle realtime changes to the chat:
    chat.onChanged((val, prev, isRemoteChange, context) => {
        if (val.title !== prev.title) { 
            alert(`Chat title changed to ${val.title} by ${isRemoteChange ? 'us' : 'someone else'}`); 
        }
    });
});

For more examples and documentation, see AceBase realtime database engine at npmjs.com

Lurcher

Answer 2

Open up a changeStream with the resumeToken. There's no guarantee of causal consistency however since we're talking multiple disparate databases.

Walcott

Answer 3

my approach was to store all data in local storage with timestamps and compare if any timestamp new and changed from the MongoDB timestamp push changed values from local storage to MongoDB in this approach you don't need to download all schemas beside other users I don't know how much sufficient it is if anybody has sufficient information and idea comment I will learn further along

this is my architecture idea:

Harvestman

Answer 4

I haven't worked with IndexDB, but the design problem isn't that uncommon. My understanding of your app is that when the client makes the connection to MongoDB, you pull a set of documents down for local storage and disconnect. The client then can do things locally (not connected to the data server), and then push up the changes.

The way I see it you've got to handle two general cases:

1. when the MongoDB server is updated and breaks continuity with the client, the client will have to
   1. poll for the data (timer?) or
   2. keep a websocket open to let notifications free-flow over the pipe
2. when the user needs to push changed data back up the pipe
   1. you can reconnect asynchronously, check for state changes, (resolving conflicts according to your business rules)
   2. have a server side (light) interface for handling conflicts (depending on complexity of your app, comparing time stamps of state changes in MongoDB to IndexedDB updates should suffice)

Tades

Do you know of anythig that might have already been impemented in this field.Our use is little simpler in the no changes are ever made locally The user can only make change requests that are pushed to the server and possibly accepted and/or rejected. If accepted they have to be pushed back to the client. So in this scenario we can never have local and remote changes on the same object. – Maryrose

there may be something in the realm of backbone.js but that is a vit out of my league. – Tades

MongoDB now supports Change Streams that allow a stream of changes to be supplied to a client program. See docs.mongodb.com/manual/changeStreams. – Treaty