In our app under development we are using Core Data with a sqlite backing store to store our data. The object model for our app is complex. Also, the total amount of data served by our app is too large to fit into an iOS (iPhone/iPad/iPod Touch) app bundle. Because of the fact that our users are, typically, interested only in a subset of the data, we've partitioned our data in such a way that the app ships with a subset (albeit, ~100 MB) of the data objects in the app bundle. Our users have the option of downloading additional data objects (of size ~5 MB to 100 MB) from our server after they pay for the additional contents through iTunes in-app purchases. The incremental data files (existing in sqlite backing stores) use the same xcdatamodel version as the data that ships with the bundle; there is zero changes to the object model. The incremental data files are downloaded from our server as a gzipped sqlite files. We don't want to bloat our app bundle by shipping the incremental contents with the app. Also, we don't want to rely on queries over webservice (because of the complex data model). We've tested the download of the incremental sqlite data from our server. We have been able to add the downloaded data store to the app's shared persistentStoreCoordinator.
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:defaultStoreURL options:options error:&error])
{
NSLog(@"Failed with error: %@", [error localizedDescription]);
abort();
}
// Check for the existence of incrementalStore
// Add incrementalStore
if (incrementalStoreExists) {
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:incrementalStoreURL options:options error:&error])
{
NSLog(@"Add of incrementalStore failed with error: %@", [error localizedDescription]);
abort();
}
}
}
However, there are two problems with doing it this way.
- Data fetch results (e.g., with NSFetchResultController) show up with the data from the incrementalStoreURL appended to the end of the data from the defaultStoreURL.
- Some of the objects are duplicated. There are many entities with read-only data in our data model; these get duplicated when we add the second persistentStore to the persistentStoreCoordinator.
Ideally, we would like Core Data to merge the object graphs from the two persistent stores into one (there are no shared relationships between data from the two stores at the time of the data download). Also, we would like to remove the duplicate objects. Searching the web, we saw a couple of questions by people attempting to do the same thing we are doing--such as this answer and this answer. We've read Marcus Zarra's blog on importing large data sets in Core Data. However, none of the solutions we've seen worked for us. We don't want to manually read and save the data from the incremental store to the default store as we think this will be very slow and error prone on the phone. Is there a more efficient way of doing the merge?
We've attempted to solve the problem by implementing a manual migration as follows. However, we haven't been able to successfully get the merge to happen. We are not really clear on the solution suggested by answers 1 and 2 referenced above. Marcus Zarra's blog addressed some of the issues we had at the outset of our project importing our large dataset into iOS.
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
NSMigrationManager *migrator = [[NSMigrationManager alloc] initWithSourceModel:__managedObjectModel destinationModel:__managedObjectModel];
if (![migrator migrateStoreFromURL:stateStoreURL
type:NSSQLiteStoreType
options:options
withMappingModel:nil
toDestinationURL:destinationStoreURL
destinationType:NSSQLiteStoreType
destinationOptions:nil
error:&error])
{
NSLog(@"%@", [error userInfo]);
abort();
}
}
It seems that the author of answer 1 ended up reading his data from the incremental store and saving to the default store. Perhaps, we've misunderstood the solution suggested by both articles 1 & 2. The size of our data may preclude us from manually reading and re-inserting our incremental data into the default store. My question is: what is the most efficient way to get the object graphs from two persistentStores (that have the same objectModel) to merge into one persistentStore?
Automatic migration works pretty well when we add new entity attributes to object graphs or modify relationships. Is there a simple solution to merging similar data into the same persistent store that will be resilient enough to stop and resume--as automatic migration is done?