Are there best practices/patterns for synchronizing local and remote data in SPA (html5 single page applications)?

Question

I'm writing a simple "todo - helloworld" with jqueryMobile + knockout + breeze + WebAPI to learn about SPAs (Single Page Application) in a mobile context (unreliable internet connection)

To enable offline usage the WebApp will leverage

• Application Cache
• Localstorage

The App should use whenever possible remote database for loading and saving data, but should be able to seamlessly switch to localstorage when offline and synchronize local/remote changes when back online.

Now back to the question: the App will use Breeze's EntityManager for managing data (local cache and remote sync)

• "remoteDb"

To mitigate inconsistency/concurrency problems I would use 2 localstorage Keys:

• "localDb" for a local copy of the remote database (list of todos)
• "localPendingChanges" for changes the App wasn't able to submit to the remote database

so the flow would more or less be (pseudocode):

LoadData
  if(online)
    load remoteDb
    save localDb                   // save a local copy of the fresh loaded remotDb
    if(localPendingChanges)
      load localPendingChanges     // we are merging in the Breeze entityManager remote data with localPendingChanges
      Savedata                     // we are online and we have pending changes, so we should sync everything back to the remoteDb as soon as possible
  if(offline)
    load localDb
    if(localPendingChanges)
      load localPendingChanges     // we are merging in the Breeze entityManager local data with localPendingChanges

SaveData
  if(online)
    save remoteDb
    clear localPendingChanges      // until we are online there is no need to keep localPendingChanges
    LoadData                       // we are loading data from remoteDb to update our localDb to the latest version
  if(offline)
    save localPendingChanges       // we are saving only the changes to localstorage

What do you think about this approach? Is it a mess? Is it ok? What about concurrency problems in a multi user scenario?

Carmacarmack

I'm looking for a good approach.Although I have the feeling there's one better, this is a very good one. +1 – Marrufo

Answer 1

This seems reasonable, not quite sure why you want two localStorage keys. The entityState (unmodified, modified, added, deleted) of every entity placed in localStorage is maintained so you can always 'extract' (see the EntityManager.getEntities method) just the pending changes from any local copy. So why not just save the entire entityManager state to localStorage before you shut down the app.

As for concurrency issues, you should definitely set a concurrency property on each entityType. As long as this exists, and if you are saving via the Entity Framework, breeze will detect any optimistic concurrency violations during a save and throw an exception. Obviously, you have determine the right behavior for your app after such an exception.

I'm sure you've seen it, but the QueryOptions.MergeStrategy (PreserveChanges, OverwriteChanges) can be very helpful in keeping or overwriting data on the local machine after any query.

Hew

I can see a reason for having a separate key (or keys) for changes. That's a reasonable way to isolate sandboxed editing sessions, especially if you can have several unsaved sandboxed edits and want them each to be dealt with in separate transactions. It also makes sense to me if you are going to store pending changes locally "as you go" to protect the user against lost work due to accidents or tombstoning. These change-sets would likely be small and short-lived; you'd want to isolate them from larger, longer-lived unchanged data. As you see, it depends on a clear understanding of app needs – Cliffcliffes

Thank You Jay, I'm using EF so I will definitelly follow your suggestion about concurrency properties (have to look a bit into it, never used EF until now :-) ) – Carmacarmack

I'll stay with the two Keys in localstorage because, although not necessary (as you pointed out), I like to keep the two separate for readability – Carmacarmack

Answer 2

I think you are correct on keeping local changes in a separate place from ones that are Sync'd with the Server. I have been tackling this problem for a few months and have come up with something that looks very much like a Version Control System where all data in a key within a set and everything is versioned separately. You can download changes from the Server into the Local database and it will handle if they have been changed on both sides through a conflict resolution callback.

At the moment I do not know much about Knockout, but the library itself is not dependant on any separate projects and is passing test cases in Node.JS, Dojo and jQuery. It has a super tight API (.get, .set, .feed (for loading downloaded data from the server) and .getFirst (for getting access to what needs to be uploaded).

The URL is at https://github.com/forbesmyester/SyncIt, it has a fairly comprehensive demo and docs too.

Transhumance