Example or explanation of Core Data Migration with multiple passes?

Question

My iPhone app needs to migrate its core data store, and some of the databases are quite large. Apple's documentation suggests using "multiple passes" to migrate data to reduce memory use. However, the documentation is very limited and doesn't explain very well how to actually do this. Can someone either point me towards a good example, or explain in detail the process of how to actually pull this off?

Outfox

did you run in memory problems actually? Is your migration leightweight or do want to use a NSMigrationManager? – Madi

Yes, the GDB console showed that there were memory warnings, and then the app crashes due to limited memory. I have tried both lightweight migration and NSMigrationManager, but right now I am trying to use NSMigrationManager. – Outfox

ok, can you go a bit more into detail what have changed? – Madi

finally, I've found out, read my answer. – Madi

Hello Jason, could you fix the like in the question? – Monseigneur

Answer 1

I've figured out what Apple hints in their documentation. It's actually very easy but a long way to go before it's obvious. I'll illustrate the explanation with an example. The initial situation is this:

Data Model Version 1

It's the model you get when you create a project with the "navigation based app with core data storage" template. I compiled it and did some hard hitting with some help of a for loop to create around 2k entries all with some different values. There we go 2.000 events with an NSDate value.

Now we add a second version of the data model, which looks like this:

Data Model Version 2

The difference is: The Event entity is gone, and we've got two new ones. One which stores a timestamp as a double and the second one which should store a date as NSString.

The goal is to transfer all Version 1 Events to the two new entities and convert the values along the migration. This results in twice the values each as a different type in a separate entity.

To migrate, we choose migration by hand and this we do with mapping models. This is also the first part of the answer to your question. We will do the migration in two steps, because it's taking long to migrate 2k entries and we like to keep the memory footprint low.

You could even go ahead and split these mapping models further to migrate only ranges of the entities. Say we got one million records, this may crash the whole process. It's possible to narrow the fetched entities down with a Filter predicate.

Back to our two mapping models.

We create the first mapping model like this:

1. New File -> Resource -> Mapping Model

2. Choose a name, I chose StepOne

3. Set source and destination data model

Mapping Model Step One

The multi pass migration doesn't need custom entity migration policies, however we will do it to get a bit more detail for this example. So we add a custom policy to the entity. This is always a subclass of NSEntityMigrationPolicy.

This policy class implements some methods to make our migration happen. However it's simple in this case so we will have to implement only one method: createDestinationInstancesForSourceInstance:entityMapping:manager:error:.

The implementation will look like this:

StepOneEntityMigrationPolicy.m

#import "StepOneEntityMigrationPolicy.h"


@implementation StepOneEntityMigrationPolicy

- (BOOL)createDestinationInstancesForSourceInstance:(NSManagedObject *)sInstance 
                                      entityMapping:(NSEntityMapping *)mapping 
                                            manager:(NSMigrationManager *)manager 
                                              error:(NSError **)error
{
    // Create a new object for the model context
    NSManagedObject *newObject = 
        [NSEntityDescription insertNewObjectForEntityForName:[mapping destinationEntityName] 
                                      inManagedObjectContext:[manager destinationContext]];

    // do our transfer of nsdate to nsstring
    NSDate *date = [sInstance valueForKey:@"timeStamp"];
    NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
    [dateFormatter setTimeStyle:NSDateFormatterMediumStyle];
    [dateFormatter setDateStyle:NSDateFormatterMediumStyle];    

    // set the value for our new object
    [newObject setValue:[dateFormatter stringFromDate:date] forKey:@"printedDate"];
    [dateFormatter release];

    // do the coupling of old and new
    [manager associateSourceInstance:sInstance withDestinationInstance:newObject forEntityMapping:mapping];

    return YES;
}

Final step: the migration itself

I'll skip the part for setting up the second mapping model which is almost identical, just a timeIntervalSince1970 used to convert the NSDate to a double.

Finally we need to trigger the migration. I'll skip the boilerplate code for now. If you need it, I'll post here. It can be found at Customizing the Migration Process it's just a merge of the first two code examples. The third and last part will be modified as follows: Instead of using the class method of the NSMappingModel class mappingModelFromBundles:forSourceModel:destinationModel: we will use the initWithContentsOfURL: because the class method will return only one, maybe the first, found mapping model in the bundle.

Now we've got the two mapping models which can be used in every pass of the loop and send the migrate method to the migration manager. That's it.

NSArray *mappingModelNames = [NSArray arrayWithObjects:@"StepOne", @"StepTwo", nil];
NSDictionary *sourceStoreOptions = nil;

NSURL *destinationStoreURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"CoreDataMigrationNew.sqlite"];

NSString *destinationStoreType = NSSQLiteStoreType;

NSDictionary *destinationStoreOptions = nil;

for (NSString *mappingModelName in mappingModelNames) {
    NSURL *fileURL = [[NSBundle mainBundle] URLForResource:mappingModelName withExtension:@"cdm"];

    NSMappingModel *mappingModel = [[NSMappingModel alloc] initWithContentsOfURL:fileURL];

    BOOL ok = [migrationManager migrateStoreFromURL:sourceStoreURL
                                               type:sourceStoreType
                                            options:sourceStoreOptions
                                   withMappingModel:mappingModel
                                   toDestinationURL:destinationStoreURL
                                    destinationType:destinationStoreType
                                 destinationOptions:destinationStoreOptions
                                              error:&error2];
    [mappingModel release];
} 

Notes

• A mapping model ends in cdm in the bundle.

• The destination store has to be provided and should not be the source store. You can after successful migration delete the old and rename the new one.

• I did some changes to the data model after the creation of the mapping models, this resulted in some compatibility errors, which I could only solve with recreating the mapping models.

Madi

Bloody hell that's complicated. What was Apple thinking? – Loyal

I don't know, but whenever I think core data is a good idea I try hard to find a simpler and more maintainable solution. – Madi

Thanks! This is a superb answer. It seems complicated, but it's not that bad once you learn the steps. The biggest problem is documentation doesn't spell it out for you like this. – Cispadane

Here's the updated link to Customizing the Migration Process. It has moved since this post was written. developer.apple.com/library/ios/documentation/Cocoa/Conceptual/… – Genethlialogy

@NickWeaver how are you determining destinationStoreURL? Are you creating it or it gets created by core data system during migration process???? – Antechamber

Hi @NickWeaver, could you please have a look at the following post. It's regarding this post which I tried to follow but was not able to get the remaining steps. #25370722 – Nichollenicholls

"because the class method will return only one, maybe the first, found mapping model in the bundle" I am very unsure about this statement. The logic of this method implies that it finds across all of the available data models in the specified bundles the one that suits metadata from the given store. This is a performant version of NSPersistentStoreCoordinator.addPersistentStore failing with an error. Migration will work w/o iterating through mapping models, you can test this if you want – Delegation

Answer 2

These questions are related:

Memory issues migrating large CoreData datastores on iPhone

Multiple Pass Core Data Migration In Chunks With iOS

To quote the first link:

This is discussed in the official documentation in the "Multiple Passes" section, however it looks like their suggested approach is to divide up your migration by entity type, i.e. make multiple mapping models, each of which migrate a subset of the entity types from the complete data model.

Mullock

Thanks for the links. The problem is that no one actually explains in detail how to set it up in multiple passes. How should I set up multiple mapping models so that it will work effectively? – Outfox

Answer 3

Suppose your database schema has 5 entities, e.g. person, student, course, class, and registration to use the standard kind of example, where student subclasses person, class implements course, and registration joins class and student. If you have made changes to all these table definitions, you have to start at the base classes, and work your way up. So, you cannot start with converting registrations, because each registration record depends on having class and students there. So, you would start with migrating only the Person table, copying existing rows into the new table, and filling in whatever new fields are there (if possible) and discarding the removed columns. Do each migration inside an autorelease pool, so that once it is done, your memory is back to start.

Once the Person table is done, then you can convert the student table over. Then hop over to Course and then Class, and the finally the Registration table.

The other consideration is the number of records, if like Person had a thousand rows, you would have to, every 100 or so, execute the NSManagedObject equivalent of a release, which is to tell the managed object context [moc refreshObject:ob mergeChanges:NO]; Also set your stale data timer way low, so that memory is flushed often.

Boiling

So are you essentially suggesting to have a new core data schema which is not part of the old schema, and copy the data to the new schema by hand? – Outfox

-1 Manually mapping your database is not necessary. You can migrate deployed databases using lightweight migration or with explicit MappingModels. – Cispadane