Core Data - mixing lightweight and custom migration

Question

So I have version 1 of my Core Data app in the App Store and now I started working on version 2.

I have some small changes to my data base model, and some custom code I need to run after those changes to complete the upgrade from version 1 to version 2.

I can use Core Data's lightweight migration to deal with the model change, and I can run my custom code after the migration is done.

The problem is, i'm not sure what will happen in the future, when I'll build version 3,4,5...

suppose this is the case:
version 1 to version 2 - use lightweight migration
version 2 to version 3 - use custom migration with model mapping
version 3 to version 4 - use lightweight migration again
and so on..

I'm not sure how to build a mechanism that deal with this mix of lightweight and custom migrations.
I couldn't find any code online or in Core Data docs that talk about this issue, I mean this is a very common issue for most of the Core Data apps out there, is there any best practice examples for this issue?

Jeffjeffcoat

Answer 1

Two methods are written below:

1. I personally like this method since it's just genius IMO, you can tell from the detail of my explanation why I'm so excited about this method. This method has been written in Marcus Zarra's core data second edition book. It's an implementation for an automatic Progressive migration process for both heavy and lightweight use.

2. A method I came up with for a possible implementation after the OP asked if there was a way to avoid having to use mapping models for each migration upgrade

Method 1 This is where Progressive Migration comes in. You create mapping models for each version upgrade that will deal between two model versions. A mapping model will tell the migration manager how to map an old version of the data store to the new version of the data store. So a mapping model requires a source and a destination.

You then have as many mapping models required that will cover all the version upgrade increments. So if you have 4 versions, youll have 3 mapping models that will aid the transition between each version upgrade. In the end youll have them named: V1toV2MappingModel, V2to3MappingModel, V3toV4MappingModel.

---

In each mapping model you can then toggle the options to let the migration manager know how you wish to migrate. Whether its via a custom policy or a normal copy migration. So you say that for v1 to v2 you require a light weight migration, simply select the appropriate mapping model, you then use the model/mapping editor to make the desired connections/mappings between old attributes to new attributes and if its a custom migration required, then go to the right mapping model, choose the product entity mapping that you wish to customise the migration for, then in the entitymapping inspector you'll see that you can apply a custom migration policy, just copy the name of your migration policy say for example MigrationPolicyV2toV3 since its that particular version that you wanted to have customised.

So in the image above you can see on the left hand side in the name of the mapping model that it's for Version 1 to Version 2. Notice that the attribute mapping for the ProductToProduct entity mapping is empty - in the middle. This is because if you look on the right of the image in the entity mapping inspector where it says Custom Policy, I've put the name of my migration policy. This then lets the migration manager know (during the migration process) how to map the attributes and values from the source to the destination - version 1 to version 2 - and it knows this because of the migration policy inputted. This also causes the value type to be changed to custom. letting you know its going to be a custom migration when it comes to the ProductToProduct entity mapping.

Then you have to define in your migration policy which will determine how you wish to copy the values over. This is where you do your custom stuff.

As you can see from the image above, its the custom migration policy I've set for the ProductToProduct entity mapping. You'll notice that I'm not actually doing anything custom, all this could have been done without the migration policy and could have been achieved by simply doing a copy migration (lightweight migration) by adjusting a few values in the entityMapping inspector and adjusting the Attribute mapping values in the image before. I only did all of this migration policy custom stuff just as an exercise so that I can learn and be prepared for the future just INCASE I ever needed to do a heavy migration. Better learn it now than later hey ;)

That's it for doing custom stuff. I suggest you read up on the following apple developer reference on NSEntityMigrationPolicy and the methods required to do more custom stuff so that you know how to have full control throughout the migration process whenever a particular revision upgrade requires some or full custom migration.

---

And for any custom/heavyweight migrations - where in my case I use a migration policy so that I can do some custom code stuff during the migration from V2 to V3 in my data store - then you create something called a 'migration policy' so that THAT mapping model will adhere to the custom migration rules your specify.

And you simply apply all the appropriate transitions/mappings for each mapping model so that the migration manager knows how to upgrade from one store to the next.

All you need then is some recursive code that will look at the existing store's meta data to determine whether it's compatible with the most current version, and if its not, it will then do a recursive migration for you, automatically, following the rules from each mapping model as it upgrades through the versions until the store is up to date with the current version.

So this way you can accommodate all users with any version and have them brought up to speed to the current version's store. So if a user is at version 1, it will recursively go from V1 to V2, then migrate to v3 all the way up to your current version. The same applies if the user has any other version.

This does mean that it will take a little longer in the migration process, but its a good way of migrating all your users no matter which version they have.

To find this progressive migration code, you need to read a book called Core Data 2nd Edition - Data storage and management for iOS, OS X, and iCloud, Chapter 3 Versioning and Migration sub chapter 3.6 Progressive Data Migration (An Academic Exercise) from pages 54 to 59.

He talks you through the code,and tells you step by step how to write the progressivelyMigrateURL:ofType:toModel:error: method. Once you've written the method with him, he then tells you how to call this method on application startup so that your users can have their stores automatically migrated progressively too.

So you should probably write the method first, then follow my steps up above or you can read through migration policies in the subchapters before.

I practically learned all this in the last 48 hours and have it all up and running now. Able to do lightweight migrations for some versions and then have custom migrations for my other versions all done automatically.

Method 2 - Another solution albeit more cumbersome IMO: You can always have lightweight migration setup bearing in mind that you apparently can achieve even complex situations with lightweight migration. But in the instances where heavy migration is required for a specific version upgrade, you can always do if statements where you can check the current store, and if and ONLY if the current version of the persistent store matches an upgrade where heavy migration is required, you then tell the migration manager to perform a heavy migration and to follow a mapping model with a migration policy for that instance only, and then resume lightweight migration if there are more version upgrades to do to get the persistent store to the most recent model version.

Lsd

+1 and thank you very much for your detailed answer. Few questions: 1. you wrote V2toV4MappingModel, it should be V2toV3MappingModel right? 2. in the 3rd paragraph you wrote that I should set light weight migration in the mapping model. what do u mean? from what I know light weight migration is done by just setting two flags in the persistent store options no? so how does one do light weight migration with mapping model? 3. i will really appreciate it if you can share your running example, or some code specifically about the mixing of light weight migration and custom migration. thank you – Jeffjeffcoat

Question 1. you wrote V2toV4MappingModel, it should be V2toV3MappingModel right? Answer: Correct. I was typing way too fast to try and help you out after I had just spent the last 58 hours on a core data problem which is when I learned all of this information that I shared with you in this answer. I wanted to get it out to you as soon as possible to save you some headache ;) lol – Lsd

Question 2. in the 3rd paragraph you wrote that I should set light weight migration in the mapping model. what do u mean? from what I know light weight migration is done by just setting two flags in the persistent store options no? so how does one do light weight migration with mapping model?. Answer: ahaha oh my. I'll have to edit my answer, again I rushed it. What I mean by that is, any lightweight stuff, i.e. not having to write any custom code or doing any complicated splitting entities into two for example, you could just sort out in the mapping editor. Thats what I meant. Cont. – Lsd

Cont. For example, if you wanted to convert the values in a fahrenheit field scaling it to Celsius you can apply an expression in the mapping editor, my point being you can do this without having to write code in a migration policy and instead can be done in the inspector pane of the mapping editor. hopefully that makes more sense. – Lsd

3. i will really appreciate it if you can share your running example, or some code specifically about the mixing of light weight migration and custom migration Did you read my answer properly? It's not code!!! it's a mixture of having mapping models in place and having classes that subclass from NSMigrationPolicy inbetween versions where you require an upgrade, and applying these policies to the mappingEntity in the appropriate mapping model that requires a custom migration, setting the Custom Policy and pointing it to your migration policy class. Cont. – Lsd

You will only need code for whenever you have a custom migration that requires migration policies to really do custom stuff and for the ProgressivelyMigrate... method which is all written out in Marcus Zarras' book in the chapter provided. Its a book thats taught me a lot in the last 50 or so hours - wherever I needed it (for the progressive migration part) - and I was up on my feet. He practically spoon feeds you the code. Here's a link to his book. pragprog.com/book/mzcd2/core-data Buy the e-book version so you can download it today. – Lsd

I think we both meaning different things when saying "lightweight migration". Correct me if i'm wrong but you see it as just a simple case of mapping model migration, and I'm talking about the technical different between lightweight and using mapping model. lightweight as described in core data docs (using only two flags when creating the psc), has performance advantages over mapping model, such as, no need to bring objects to memory and copy them. So I want to take advantage of this method when possible and not use mapping model for each migration even when not required. – Jeffjeffcoat

and regarding the book you mention, well it will take some time to order it and it cost over 40$ to purchase and ship it to me. so I cant really use it as reference to the code. I would appreciate it if you can share the piece of code you were talking about. – Jeffjeffcoat

You can get an electronic version of the book for much less than $40.00 and there is no shipping. Seems like you have spent more time here on SO asking this question than the value of the book ;-) – Mustee

@Hi eyal, I contacted Marcus S Zarra directly to ask for permission to transfer HUGE chunks of code. I fet uncomfortable copying 3 pages just like that. I hope you understand. There, now you have the author himself putting a close on the situation! I have tried to help you as much as I can Eyal, I hope that you will do your part. – Lsd

Hey @MarcusS.Zarra, using the method I used in your Progressive Migration - (Academic Exercise) sub chapter, is it now possible to say that for version 4-5 just do a lightweight migration? Atm, I created a mapping model named MappingModelV4toV5 even though there is no custom code. Someone tells me that I'm bringing everything into memory for THIS part of the migration from V4toV5 when that only requires a lightweight migration. I explained that with Marcus' setup, a mapping model is required for each version upgrade` lightweight or not otherwise the code breaks is that correct? Cont. – Lsd

Cont. From version 1 to 4, Its been custom migrations, with custom policies that were required so your chapter was awesome to allow for a smooth migration operation where I was in control. For V4toV5, lightweight isnt required. Is it possible to avoid using a mapping model for that part of the migration so that I can save a bit of time during migration since it doesnt need to be all copied into memory - which I understand is what happens when using a mapping model. I ask because Eyal has the same question. – Lsd

@MarcusS.Zarra I will be glad to purchase your book if its talking about this issue. As I asked Pavan I'm not sure this solution of using mapping model for every migration even if not necessary is what i'm looking for. Of course I have no intention to harm you work :) – Jeffjeffcoat

@Jeffjeffcoat based on your requirements, I have provided an adequate solution and what is normally the go-to method with a situation like yours. I'm not sure what else you're expecting or what else we can offer you based on your specific requirements but this is one of the least confusing ways to go about it especially when you want to incorporate heavy migration too. Another solution albeit more cumbersome: You can always have lightweight migration setup bearing in mind that you apparently can achieve even complex situations with lightweight migration. But in the instances where heavy ... Cont. – Lsd

Cont. But in the instances where heavy migration is required for a specific version upgrade, you can always do if statements where you can check the current store, and if and ONLY if the current version of the persistent store matches an upgrade where heavy migration is required, you then tell the migration manager to perform a heavy migration and to follow a mapping model with a migration policy for that instance only, and then resume lightweight migration if there are more version upgrades to do to get the persistent store to the most recent model version. – Lsd

@Lsd I think you are right, I probably should find a combination of this approach when necessary and pure lightweight when not. Thank you for all your effort and help:) – Jeffjeffcoat

Let us continue this discussion in chat. – Lsd

@Lsd Great explanation! But I have a question to your Method 2: where exactly in Marcus Zarras progressive migration process can I add the if check for lightweight migration? As I understand it correctly, the only way to trigger the lightweight migration is using the method addPersistentStoreWithType: with the 2 options flag. How and from where can I detect that I now need to do custom migration after having done the lightweight migration? And after doing heavy migration, it needs to continue with light migration again. cont' – Horribly

cont' Example: Model1 > Model2 (light) > Model3 (heavy) > Model4 (light). User is still at Model1 and now starts app and it needs to migrate to latest version. – Horribly

@Marcus S. Zarra and Pavan Suppose I have 3 models now all performing light-weight migration or only v2 to v3 performing heavy-weight migration, then also I have to write the progressiveMigrationCode? Will core-data itself not handles progressive migrations? – Imaimage

No, core data does not, will not, perform a progressive migration. There is no concept of v1 -> v2 -> v3 in Core Data. There is only source -> destination. You need to TEST for each possible migration and make sure you handle them. – Mustee

Answer 2

If you can refetch data from a server or anything like that, the best way to deal with this is by removing your old model and immediately recreating a new one with the current structure. This is embedded in a singleton class and is called every time a new managedobjectcontext has to be created. My code for this is the following:

- (NSManagedObjectContext *)managedObjectContext {
    if (__managedObjectContext != nil) {
        return __managedObjectContext;
    }

    NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
    if (coordinator != nil) {
        __managedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
        [__managedObjectContext setPersistentStoreCoordinator:coordinator];
    }
    return __managedObjectContext;
}

- (NSPersistentStoreCoordinator *)persistentStoreCoordinator {
    if (__persistentStoreCoordinator != nil) {
        return __persistentStoreCoordinator;
    }

    NSURL *storeURL = [[[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject] URLByAppendingPathComponent:@"YOURDB.sqlite"];

    NSError *error = nil;
    __persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
    if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:nil error:&error]) {
        if (error.code == 134100) {
            if ( [[NSFileManager defaultManager] fileExistsAtPath: [storeURL path]] ) {
                NSDictionary *existingPersistentStoreMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType: NSSQLiteStoreType URL: storeURL error: &error];
                if ( !existingPersistentStoreMetadata ) {
                    // Something *really* bad has happened to the persistent store
                    [NSException raise: NSInternalInconsistencyException format: @"Failed to read metadata for persistent store %@: %@", storeURL, error];
                }

                if ( ![[self managedObjectModel] isConfiguration: nil compatibleWithStoreMetadata: existingPersistentStoreMetadata] ) {
                    if (![[NSFileManager defaultManager] removeItemAtURL: storeURL error: &error] ) {
                        NSLog(@"*** Could not delete persistent store, %@", error);
                        abort();
                    } else {
                        [__persistentStoreCoordinator addPersistentStoreWithType: NSSQLiteStoreType configuration: nil URL: storeURL options: nil error: &error];
                    }
                }
            }
        } else {
            NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
            abort();
        }
    }

    return __persistentStoreCoordinator;
}

Duodecimo

This is a good idea especially if you can fetch the data from the server. But if there's thousands of rows of data that needs to be fetched then the user will have to wait a few minutes for all the data to arrive if on a 3g connection. I have an app running and my client has to wait that long, thats why I incorporated Core Data so that he wouldnt need to fetch the data each and every time the store is upgraded. I think your method will work fantastically well when there isn't a lot of data to be fetched from the server AND if you're not saving any user data (unless its stored on the server 2) – Lsd

Answer 3

The simplest way is keeping old attributes in new version and not using them anymore. if you need map them to new attributes, just do the mapping after lightweight DB upgrade. The world will keep peaceful.

The benefits:

1. no need to make mapping model.
2. keep using light weight upgrade.

---

Let me give you example: let's say we have a entity named Student with attribute name in V1, and we gonna map it to a new attribute firstName in V2. and we have a method as below, so then we can call this method after regular lightweight CoreData upgrade, enjoy!

- (void)migrateStudentRecords {
    NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] initWithEntityName:@"Student"];
    NSArray *result = [defaultContext executeFetchRequest:fetchRequest error:error];
    if (result && result.count > 0) {
        [result enumerateObjectsUsingBlock:^(Student  *_Nonnull newRecord, NSUInteger idx, BOOL * _Nonnull stop) {
            newRecord.firstName = newRecord.name;
            newRecord.name = nil;
        }];
        if (defaultContext.hasChanges) {
            [defaultContext save:error];
        }
    }
}

Shadbush