Migrating data when iCloud is turned on/off

Question

The local account

From the WWDC 2013 207 session about Core Data and iCloud:

You provide us a single store URL inside the application's local sandbox and we then create an opaque container with an entry inside of it for each account on the system, including the local account, which is our term for what happens when there is no iCloud account on the system. This is a special store that's managed by Core Data so that you don't have to do anything special because your user doesn't have an iCloud account.

In iOS 7/OS X 10.9, Core Data with iCloud will automatically use a local account for situations in which iCloud is off. Unlike the fallback store (used when iCloud is on but unreachable), the local account will be wholly replaced by an iCloud account when the service is on, without any merging. The data in the local account is only accesible if iCloud is off. This happens when:

• There is no iCloud account.
• There is an iCloud account, but "Documents & Data" has been disabled.
• There is an iCloud account, but the app has been disabled in "Documents & Data".

The above is what I understand from experimentation. Please correct me if I'm wrong.

When data disappears

Used as is, the local account user experience is awful. If you add data to an app with iCloud off and then turn it on, the data will "disappear" and you might think that it has been deleted. If you add data to an app with iCloud on, and then turn it off, the data will also "disappear".

I have seen examples that try to work around this by adding (more) iCloud settings to the app and managing their own "local" store (not the one provided by iCloud). This reeks of duplicating work to me.

Leveraging the local account for data migration

How about this approach?

• Always use Core Data and iCloud, no matter if iCloud is on or off.
• When iCloud goes from off to on, ask users if they want to merge the local account with the iCloud account. If yes, merge, remove duplicates prioritizing local and empty the local account.
• When iCloud goes from on to off, ask users if they want to merge the iCloud store with the local account. If yes, merge and remove duplicates prioritizing iCloud.

This is similar to what Reminders does. However, Reminders asks the user about data migration directly from iCloud settings, which is something that us developers can't do.

Questions

1) Does this approach have any drawbacks or border cases that might not be obvious at first glance? Maybe we're not meant to use the iCloud-generated local account like this.

2) Are NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification sufficient to detect all the possible on to off and off to on iCloud transitions?

3) Would you do the user prompt and merging between NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification, or gather all the information in those and wait until the store is changed? I ask because these notifications appear to be sent in background, and blocking them to perform a potentially long operation might not be what Core Data expects.

Senary

Good luck with this. I spent a week trying to accomplish similar to what you have outlined. And at the end, I could not come up with a reliable solution that can guarantee user will not loosing any data. I just used a local store and iCloud only to backup the store. – Lycia

@Lycia Would you mind posting the problems you faced? "It's a bad idea because..." is also a valid answer. – Senary

The correct notification for iCloud account changes is NSUbiquityIdentityDidChangeNotification, however you can rely on getting this notification if your app is not running. NSPersistentStoreCoordinator notifications are not related only to iCloud transitions. For a more detailed explanation see the link in my answer below. There is a pretty detailed explanation on the UIManagedDocument & iCloud Integration page here ossh.com.au/design-and-technology/software-development/… – Gillenwater

Am I the only one who thinks the complexity of Apple's solution is ridiculous? I tried in vain to adopt Apple's solution for about a year leading up to iOS 7. Never worked, and even when I could get close, the API is so crazily complex. My advice is to take a look at other options including Wasabi Sync and Ensembles. They just merge cloud and local data, which is usually what the user expects. (Disclosure: I develop Ensembles) – May

@DrewMcCormack I gave Ensembles a look. Thanks for open sourcing it. Given your experience, would you mind giving my answer a look to see if you can spot any problems? – Senary

Answer 1

I will attempt to answer my own question, partly to organise my thoughts and partly to reply to @DuncanGroenewald.

1) Does this approach have any drawbacks or border cases that might not be obvious at first glance?

Yes. The local and iCloud account stores are managed by Core Data and can be removed at any time.

In practice, I don't think the local account store will be ever removed as it cannot be recreated from iCloud.

Regarding iCloud account stores, I can see two scenarios in which they might be removed: a) to free space after the user turned iCloud off or b) because the user requested it by selecting Settings > iCloud > Delete All.

If the user requested it, then you might argue that data migration is not a concern.

If if was to free space, then yes, it's a problem. However, the same problem exists in any other method as your app is not woken up when iCloud account stores are removed.

2) Are NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification sufficient to detect all the possible on to off and off to on iCloud transitions?

Yes. It requires you to always create the persistent store with NSPersistentStoreUbiquitousContentNameKey, no matter if iCloud is on or off. Like this:

[self.managedObjectContext.persistentStoreCoordinator
 addPersistentStoreWithType:NSSQLiteStoreType
 configuration:nil
 URL:storeURL
 options:@{ NSPersistentStoreUbiquitousContentNameKey : @"someName" }
 error:&error];

In fact, only listening to NSPersistentStoreCoordinatorStoresDidChangeNotification is enough (as shown below). This will be called when the store is added at startup or changed during execution.

3) Would you do the user prompt and merging between NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification, or gather all the information in those and wait until the store is changed? I ask because these notifications appear to be sent in background, and blocking them to perform a potentially long operation might not be what Core Data expects.

This is how I would do it in NSPersistentStoreCoordinatorStoresDidChangeNotification.

Since this notification is sent both at startup and when the store changes during execution, we can use it to save the current store url and ubiquity identity token (if any).

Then we check if we are in a on/off transition scenario and migrate data accordingly.

For brevity's sake, I'm not including any UI code, user prompts or error handling. You should ask (or at the very least inform) the user before doing any migration.

- (void)storesDidChange:(NSNotification *)notification
{
    NSDictionary *userInfo = notification.userInfo;
    NSPersistentStoreUbiquitousTransitionType transitionType = [[userInfo objectForKey:NSPersistentStoreUbiquitousTransitionTypeKey] integerValue];
    NSPersistentStore *persistentStore = [userInfo[NSAddedPersistentStoresKey] firstObject];
    id<NSCoding> ubiquityIdentityToken = [NSFileManager defaultManager].ubiquityIdentityToken;

    NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
    if (transitionType != NSPersistentStoreUbiquitousTransitionTypeInitialImportCompleted) { // We only care of cases if the store was added or removed
        NSData *previousArchivedUbiquityIdentityToken = [defaults objectForKey:HPDefaultsUbiquityIdentityTokenKey];
        if (previousArchivedUbiquityIdentityToken) { // Was using ubiquity store
            if (!ubiquityIdentityToken) { // Changed to local account
                NSString *urlString = [defaults objectForKey:HPDefaultsPersistentStoreURLKey];
                NSURL *previousPersistentStoreURL = [NSURL URLWithString:urlString];
                [self importPersistentStoreAtURL:previousPersistentStoreURL
                 isLocal:NO
                 intoPersistentStore:persistentStore];
            }
        } else { // Was using local account
            if (ubiquityIdentityToken) { // Changed to ubiquity store
                NSString *urlString = [defaults objectForKey:HPDefaultsPersistentStoreURLKey];
                NSURL *previousPersistentStoreURL = [NSURL URLWithString:urlString];
                [self importPersistentStoreAtURL:previousPersistentStoreURL 
                 isLocal:YES 
                 intoPersistentStore:persistentStore];
            }
        }
    }
    if (ubiquityIdentityToken) {
        NSData *archivedUbiquityIdentityToken = [NSKeyedArchiver archivedDataWithRootObject:ubiquityIdentityToken];
        [defaults setObject:archivedUbiquityIdentityToken forKey:HPModelManagerUbiquityIdentityTokenKey];
    } else {
        [defaults removeObjectForKey:HPModelManagerUbiquityIdentityTokenKey];
    }
    NSString *urlString = persistentStore.URL.absoluteString;
    [defaults setObject:urlString forKey:HPDefaultsPersistentStoreURLKey];
    dispatch_async(dispatch_get_main_queue(), ^{
        // Update UI
    });
}

Then:

- (void)importPersistentStoreAtURL:(NSURL*)importPersistentStoreURL 
    isLocal:(BOOL)isLocal 
    intoPersistentStore:(NSPersistentStore*)persistentStore 
{
    if (!isLocal) { 
        // Create a copy because we can't add an ubiquity store as a local store without removing the ubiquitous metadata first,
        // and we don't want to modify the original ubiquity store.
        importPersistentStoreURL = [self copyPersistentStoreAtURL:importPersistentStoreURL];
    }
    if (!importPersistentStoreURL) return;

    // You might want to use a different concurrency type, depending on how you handle migration and the concurrency type of your current context
    NSManagedObjectContext *importContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
    importContext.persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:self.managedObjectModel];
    NSPersistentStore *importStore = [importContext.persistentStoreCoordinator
            addPersistentStoreWithType:NSSQLiteStoreType
            configuration:nil
            URL:importPersistentStoreURL
            options:@{NSPersistentStoreRemoveUbiquitousMetadataOption : @(YES)}
            error:nil];

    [self importContext:importContext intoContext:_managedObjectContext];
    if (!isLocal) {
        [[NSFileManager defaultManager] removeItemAtURL:importPersistentStoreURL error:nil];
    }
}

The data migration is performed in importContext:intoContext. This logic will depend of your model and duplicate and conflict policies.

I can't tell if this might have unwanted side effects. Obviously it might take quite a while depending on the size and data of the persistent store. If I find any problems I will edit the answer.

Senary

The problem is you can't ever guarantee that the store file will actually be there when you try and do the migration. So what is the app going to do when it is not there ? – Gillenwater

And how would you give the user the ability to actually remove content from iCloud? You would need to use the removeUbiquitousContentAndPersistentStoreAtURL API and I assume this will only work while the user is logged in to iCloud. Using a Settings bundle allows you to move content locally and then delete everything from iCloud before logging out of iCloud. Using the security settings only does not. – Gillenwater

Gracefully inform the user that no data was lost and they need to turn iCloud back on to see. This is only a (potential) problem when they switch iCloud off, as the local account store must always be there, as it's not iCloud-backed. If the iCloud account stores go away, there's not much either your solution or my solution can do. Or does yours work around this somehow? – Senary

In fact given that when switching Documents & Data off the user will get a message telling them that all iCloud data will be removed from the device wouldn't they then be confused because the iCloud data is mysteriously still available. And worse, when they turn iCloud back on, you would have to deal with de-duplication of essentially the entire store or the user would have to be asked if they want to merge the local data with whatever is in iCloud already or whether they want to use the iCloud data or whether to replace the iCloud data with whatever is local. Not without some challenges – Gillenwater

Answering in order. "how would you give the user the ability to actually remove content from iCloud?" This can be a custom Setting yes. I'm not against this. I'm just against adding a custom setting for turning iCloud on and off, as it already exists. – Senary

"wouldn't they then be confused because the iCloud data is mysteriously still available." I didn't include UI and user prompts in the code for brevity's sake. Of course, I would use the same messages that Apple uses in iCloud settings to inform the user, and give them a choice to migrate (or not). If they chose to migrate, yes, you have to deal with duplicates but you would have the same problem in any solution, wouldn't you? – Senary

My approach I always tell the user that iCloud data is not accessible if iCloud gets turned off in security settings (or the user logs out). Its a consistent experience each time and its also consistent with what other apps do (well I have not tested them all !). If they choose to stop using iCloud then I ask if they want to move the iCloud data to local and then I remove the iCloud data - again not confusing, it will even automatically disappear from other devices. – Gillenwater

Your approach you can't remove the iCloud content (using removeUbiquitousContentAndPersistentStoreAtURL API) because the iCloud container is no longer available when you get the storesDidChange notification so you have to come up with some other way to do that. Either way there is no simple solution. – Gillenwater

let us continue this discussion in chat – Senary

we should write up all these options and the implications of the different choices - different approach may suit different situations. WRT you prompting the user - you could not ask them if they want to "Keep using iCloud" - because you can't turn iCloud back on from your App. I guess you could tell them to go back and turn the security settings back on. Still how would you deal with deleting all the iCloud content if they chose the "Keep on my iPhone" option. – Gillenwater

@hpique: What are your experiences today? I'm specially interested in issues with migration from local to iCloud and vice versa. – Isomer

Answer 2

I think you have misunderstood what was said in the 207 session.

Core Data will not automatically create a local and an iCloud store for you, well not ones that will synchronise data when the iCloud account if turned off anyway. Depending on what the user has selected you have to create the store either using the NSPersistentStoreUbiquityNameKey option (for an iCloud store) or not using it (for a local store).

Because the default security setting for a new apps Data&Documents is ON when your app is first installed you MUST ask the user if they want to use iCloud or not. Try it out with Apple's Pages app.

If the user subsequently changes the preference setting your App must migrate the store to or from iCloud.

The part Core Data handles automatically is if you switch the iCloud Account (log out and log in with a different account) then the App will run with whatever Core Data store might have been created while logged in to this account.

See the transcript below where it quite clearly states that the iCloud store gets removed when the account goes away. It's gone, kaput, a dead parrot. So while you get a chance to save only the change logs remain locally in case the account gets used again in future.

You simply implement your will change handlers and respond to NSPersistentStoreCoordinator Stores Will Change and will notify you automatically when we need to change the persistent store file because there's new account on the system.

Of course, you can then call NSManagedObjectContext save and NSManagedObjectContext reset.

Now once you've done that, we'll remove the store from the coordinator just as with the asynchronous setup process and then we'll send you NSPersistentStoreCoordinator Storage Did Change notification, again, just like asynchronous setup and you can begin working with your application as you normally would.

Now, let's talk about this in a little bit more detail.

When you receive NSPersistentStoreCoordinator Stores Will Change notification, the persistent store is still available to use, and so unlike what we advised you of last year where you had to immediately drop the persistent store and wipe out your managed object context, you can still write to the managed object context and those changes will be persistent locally to be imported to the account if it every comes back.

This means that although your user's changes won't make it to iCloud immediately, if they ever sign in again, they'll be there and waiting.

Finally, all of these store files will be managed by Core Data and that means that we could remove them at any time.

Each store will be removed once its account has gone away because we can rebuild the file from the cloud.

So we want to free up as much disk space as possible for your application to use and not have old store files lying around that could take up additional resources.

and a bit further on

We're also introducing a new option to help you create backups or local copies of the iCloud persistent store called NSPersistentStore Remove Ubiquitous Metadata Option.

This removes all associated metadata from the iCloud store; that means, anything that we write into the metadata dictionary as well as the store file itself, and it's critical if you want to use the migration API to create backups or local copies at a persistent store you wish to open without the iCloud options.

Also take a look at this link to the errata for Tim Roadley's book

http://timroadley.com/2014/02/13/learning-core-data-for-ios-errata/

If you are logged in to iCloud and then the user changes the app preference setting (not the same as the Data&Documents security setting) to turn iCloud off your App should then ask the user if they want to migrate the existing iCloud store to the local (again - try this with Pages and see what messages you get).

I have posted a sample app that does all of this here. Take a look at the video to see the expected behaviour. http://ossh.com.au/design-and-technology/software-development/

Some of the features of the sample apps include:

Features include:

• Sample iOS and OSX Core Data Apps with iCloud Integration
• Use of Local or iCloud Core Data store
• Includes a Settings Bundle (note that this creates a settings page in the Settings App) that includes:
  • Use iCloud preference setting (ON or OFF)
  • Make Backup preference setting (ON or OFF)
  • Display application Version and Build Number
• Prompts the user about storage options when the Use iCloud preference is changed to ON
• Migrates Core Data store to and from iCloud depending on the users preference setting and response to prompts
• Detects deletion of iCloud store from another device and cleans up by creating a new empty iCloud store
• Checks for existing iCloud files when migrating local store to iCloud and prompts user whether to merge or discard data in local store if an iCloud file exists
• Makes a Backup of the Core Data store if Make Backup preference is set to ON.  Backup file name is persistentStore_Backup_yyyy_MM_dd_HH_mm_ss. To use it:
  • set Backup preference ON and next time the app is activated it will make a backup of the current Core Data store and reset the preference to OFF
  • file can be copied to PC or Mac from iTunes
  • to restore simply set app to use Local files (Use iCloud preference OFF) and replace the persistentStore file with the required backup file (note the file must be called persistentStore).
• Editing record and save/cancel edits in detailed view
• Asynchronous opening of Core Data store to ensure long migrations don't block the main thread and cause App to be terminated
• Loading of data on background thread with Pull to Refresh in main UITableView to start another background thread (you can start multiple background threads running simultaneously, take care!) 
• Display related objects in detailView using UITableView, fetchedResultsController and predicate to filter selection
• Load Seed Data if a no store exists already, checks if iCloud file has been created by another device
• iCloud Upload/Download Status indicator, network activity indicator turns on when Core Data transaction logs need to be synced, are busy syncing, being imported or when background tasks are running
• Sidebar style UI with multiple master and detail views for both iOS and OS X apps
• Backup File Manager which allows you to make backups, copy backup files to and from iCloud, send and receive backup files via email and restore from a backup file.

Gillenwater

I'll quote the WWDC talk: "You provide us a single store URL inside the application's local sandbox and we then create an opaque container with an entry inside of it for each account on the system, including the local account, which is our term for what happens when there is no iCloud account on the system. This is a special store that's managed by Core Data so that you don't have to do anything special because your user doesn't have an iCloud account." This is consistent with my tests. – Senary

Regarding Pages, I'd argue that confirming iCloud usage makes more sense for document-based apps than data-based apps, where users might be less concerned about the current location of the data. Reminders might be a more appropriate example. However, because it's an Apple app, they handle migration and user prompts from iCloud settings which is a bit unfair as we can't do that. – Senary

I have edited the question to reflect this discussion. Thanks for your input. – Senary

And I have also added some extracts from that session. – Gillenwater

Oh and as for the 'Use iCloud' settings - that is straight from Apple too. I added a link to Tim Roadley's errata which is also useful as he explains the rationale for why the Settings Bundle needs to be used. – Gillenwater

Thanks again @DuncanGroenewald. I still think you can leverage the local account managed by iCloud without the need of additional settings. I'll write it down as an answer shortly, as comments are too restrictive. BTW, your sample code and Tim Roadley's book were extremely helpful to me. – Senary