CoreData with CloudKit: After model migration everything is duplicated

Question

I tried to migrate my CoreData-Model (with CloudKit) and it duplicated all of the objects I had stored. How can I correctly migrate when using CoreData with CloudKit?

Summary

I am using CoreData with CloudKit. A few days ago, I made some changes to my model and therefore needed to migrate. This is how it went (see below for details):

1. I just made the changes in my model (Model.xcdatamodel), without changing the version of the model and installed it on my iPhone for testing -> Crash with message "Cannot migrate store in-place: constraint violation during attempted migration".

2. I created a new version of the model (Model 2.xcdatamodel) and made the changes there. I then created a .xcmappingmodel to manage the migration. No crash & it worked, however...

3. All entries in my app are now duplicated, which of course was not as intended.

What I changed in the model:

My original (source) model had two entities A and B. There is a many-to-many mapping between A and B. I did the following changes.

• Add two new entities C and D, with one data field ("name")
• Create a 1-to-many mapping between each of the two new entities C, D and one of my existing ones (A)

I did just create the .xcmappingmodel-file and not change anything in it. For the existing entities A and B it has the entries to take over the previous data, like this:

destination attribute: name
value expression: $source.name

For the existing mapping A-B (entity B is called "Tag") it has: FUNCTION($manager, "destinationInstancesForEntityMappingNamed:sourceInstances:" , "TagToTag", $source.tags) And similar for the inverse relationship.

How I build my CoreData stack

I followed the documentation from Apple. My code looks like this (I made a CoreDataManager-class):

[...]
lazy var persistentContainer: NSPersistentContainer = {
    let container: NSPersistentContainer
    container = NSPersistentCloudKitContainer(name: containerName)
    let storeDescription = container.persistentStoreDescriptions.first
    storeDescription?.type = NSSQLiteStoreType

    container.loadPersistentStores { (_, error) in
        if let error = error as NSError? {
            fatalError("Unresolved error when loading CoreData persistent stores: \(error), \(error.userInfo)")
        }
    }
    return container
}()

lazy var mainContext: NSManagedObjectContext = {
    let context = self.persistentContainer.viewContext
    context.automaticallyMergesChangesFromParent = true

    return context
}()
[...]

I don't really know what I did wrong or how I can fix this. Would appreciate if anybody can point me in the right direction.

Niccolo

I have same problem😢 – Cornetist

and, I did some investigations... 1. There are duplicated records in the actual sqlite file. The one is migrated record, and the other is old schemed record. 2. The duplication is not occurred when I don't use xcmappingmodel file. – Cornetist

any solutions ? – Marroquin

Answer 1

We have the same issue and it seems to occur EVERY time we do a heavy weight model migration using a Mapping Model. After much research, it appears that you/we are NOT doing anything wrong.

Apple CloudKit Documentation seems to suggest that data duplication is by design so that older versions of the App accessing Core Data will be able to use older data. This might be the reason why Unique Constraints are not allowed in CloudKit (we are not sure). See this Apple CloudKit Docs. Note the section titled: Update The Production Schema and the recommendation to add a version number to Core Data Entities.

• Incrementally add new fields to existing record types. If you adopt this approach, older versions of your app have access to every record a user creates, but not every field.

• Version your entities by including a version attribute from the outset, and use a fetch request to select only those records that are compatible with the current version of the app. If you adopt this approach, older versions of your app won’t fetch records that a user creates with a more recent version, effectively hiding them on that device.

Therefore we concluded that the only solution was to do one or both of the following :

1. Version our Core Data Entities and have newer versions of the software filter older model entities out during the fetch request.
2. Write Code to manually remove the older records, but this might have problematic consequences for users running older versions of the software on devices that are unable to update because of OS incompatibility.

There are some solutions out there for using NSPersistentHistoryTrackingKey but these seem quite complicated and it was not clear to us what advantage there was to the extra complexity vs. simply writing some simple de-dupe code using the UUID we fortunately did have in our Model Entities.

That code can simply be called each time the App starts up and initializes Core Data. We do it each time because our software runs on both MacOS and iOS devices and shares data between all the versions and there is no way to know when some other device will upgrade to the newer data model and duplicate data. The code to de-dupe was simply this:

func deDuplicateAfterMigrationFrom14to15()
{
    print("**** CoreDataUtil DeDupe UUIDs without new attribute or too many with new")
    let moc = pc.viewContext
    let chartsFetch = NSFetchRequest<NSFetchRequestResult>(entityName:"Charts")     // Fetch all charts
    do {
        let fetchedCharts = try moc.fetch(chartsFetch) as! [Chart]
        for chart in fetchedCharts
        {
            // Find and Remove duplicate UUID
            chartsFetch.predicate = NSPredicate(format:"uuid == %@", chart.uuid)
            do {
                let fetchedChartsWithUUID = try moc.fetch(chartsFetch) as! [Chart]
                if(fetchedChartsWithUUID.count > 1) {
                    for(index, chartWithUUID) in fetchedChartsWithUUID.enumerated() {
                        // Find old Entity without new attribute
                        let nameFirstChar = chartWithUUID.nameFirstChar ?? ""
                        if(nameFirstChar.isEmpty) {
                            print("***** DeDupe OLD Chart UUID-> " + chartWithUUID.uuid + " NAME[\(index)]-> " + chartWithUUID.name)
                            self.pc.viewContext.delete(chartWithUUID)
                        }
                        // Find extra copies of updated Entity created by other devices.
                        else if(!nameFirstChar.isEmpty && index > 1) {
                            print("***** DeDupe NEW Extra Chart UUID-> " + chartWithUUID.uuid + " NAME[\(index)]-> " + chartWithUUID.name)
                            self.pc.viewContext.delete(chartWithUUID)
                        }
                    }
                    try moc.save()
                }
            }
            catch {
                print("****** De-Dupe UUID Failed for UUID?: \(error)")
            }
        }
        try moc.save()
    }
    catch {
        print("****** De-Dupe initial Fetch failed: \(error)")
    }
    print("**** CoreDataUtil De-Dupe DONE")
}

Henton

Answer 2

My impression is, that a soon as you use a mapping as part of the migration there are created new NSManagedObjects in CoreData and the original ones are deleted. When syncing with CloudKit this leads to a duplication: CloudKit sends back the old instances, while CoreData is uploading the migrated ones.

One could distinguish those instances using a version number, but question for me is when to clean this up? I'd like to do it while setting up the CoreData stack, but it takes some time after the migration for the duplication to happen and I suppose it might take even longer in case there does not take place an CloudKit sync (not network, ...). But I haven't checked that yet.

Dupuy