Best strategy for synching data in iPhone app

Question

I am working on a regular iPhone app which pulls data from a server (XML, JSON, etc...), and I'm wondering what is the best way to implement synching data. Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access and flexibility (adaptable when the structure of the database changes slightly, like a new column). I know it varies from app to app, but can you guys share some of your strategy/experience?

For me, I'm thinking of something like this:

1) Store Last Modified Date in iPhone

2) Upon launching, send a message like getNewData.php?lastModifiedDate=...

3) Server will process and send back only modified data from last time.

4) This data is formatted as so:

<+><data id="..."></data></+> // add this to SQLite/CoreData

<-><data id="..."></data></-> // remove this

<%><data id="..."><attribute>newValue</attribute></data></%> // new modified value

I don't want to make <+>, <->, <%>... for each attribute as well, because it would be too complicated, so probably when receive a <%> field, I would just remove the data with the specified id and then add it again (assuming id here is not some automatically auto-incremented field).

5) Once everything is downloaded and updated, I will update the Last Modified Date field.

The main problem with this strategy is: If the network goes down when I am updating something => the Last Modified Date is not yet updated => next time I relaunch the app, I will have to go through the same thing again. Not to mention potential inconsistent data. If I use a temporary table for update and make the whole thing atomic, it would work, but then again, if the update is too long (lots of data change), the user has to wait a long time until new data is available. Should I use Last-Modified-Date for each of the data field and update data gradually?

Offish

Answer 1

I would start by making the update routine atomic, since you'll have enough on your hands figuring out how to get the client-server communication working properly.

After that is a good time to consider tweaking it to be incremental, but only after you do some testing to figure out if it's really necessary. If you're tuning your update protocol to be as low bandwidth as possible, you might discover that even a "big" update is downloaded fast enough.

Another way to look at it is to ask yourself, how often is there going to be network trouble when an average user is doing a sync? You probably don't want to tune for unlikely scenarios.

If you are trying to optimize (minimize) the data transfer you may want to consider a different format than XML, since XML is fairly verbose. Or at least you may want to trade in XML readability for space by making each element name and attribute as small as possible, and eliminate all unnecessary whitespace.

Lateen

Answer 2

Your basic scheme is good. The thing you need to do is to somehow make your updates idempotent so that you can restart a partially-completed transfer without risk. This is a better way to go than to try to implement some sort of true atomic commit (though you could do that too, using, eg, the SQLite database).

In our experience fairly large updates (10s of KB) can be downloaded quite rapidly, if the server is fast enough. No great need to break updates up into tiny bits. But certainly it won't hurt to try to minimize the amount of data transferred by keeping more granular info on "last update".

(And definitely you should use JSON rather than XML as your transmitted data representation.)

Lakendra

Answer 3

Wonder if you have considered using a Sync Framework to manage the synchronization. If that interests you can take a look at the open source project, OpenMobster's Sync service. You can do the following sync operations

• two-way
• one-way client
• one-way device
• bootup

Besides that, all modifications are automatically tracked and synced with the Cloud. You can have your app offline when network connection is down. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization like iCloud across multiple devices

Also, modifications in the Cloud are synched using Push notifications, so the data is always current even if it is stored locally.

In your case,

Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access

• Speed: Only the changes are sent across the network in both directions

• Robustness: It stores data in a transactional store like sqlite and any failed updates are communicated in the SyncML payload. Only the successful operations are processed while the failed operations are re-tried during the next sync

Here is a link to the open source project: http://openmobster.googlecode.com

Here is a link to iPhone App Sync: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp

Stark