Out-Of-Memory while doing Core Data migration

Question

I'm migrating a CoreData model between two versions of an application. I was storing binary data as blobs in the previous version and I want to take them out of the blobs for performance. My issue is that during the migration it seems that Core Data loads everything into memory which leads to Low Memory Warnings and then to my app being killed.

Apple documentation suggests the following : http://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/vmCustomizingTheProcess.html#//apple_ref/doc/uid/TP40005510-SW9

However, it seems to rely on the fact that the large objects are applied different mapping. In my case, all the objects are basically the same and the same mapping has to be applied to each of them. I don't see in this case how I could apply their technique.

How should I handle a migration with very large objects ?

Veinstone

Answer 1

I'm guessing that you have a bunch of changes you want to make in addition to pulling the data out of blobs. My suggestion is to do the migration in a few stages. I'm kind of thinking out loud here, so it might be possible to improve on this. This requires you to be using SQLite.

To make this work, you're going to have three versions of your model:

1. The original model
2. The model with the attribute removed (and possibly with a special unique ID added--see below)
3. The model with all of the changes you've made, including the addition of the new entity and relationships replacing the attribute

The reason to do this is that the transition from version 1 to 2 should be doable with an automatic lightweight migration. In that case Core Data doesn't need to load anything into memory--it just issues SQL statements to make the changes directly on the database.

So, you start by setting up your persistent store coordinator using the old model version. Once you've loaded the data, go through all of the objects you're migrating, extract the binary attribute, and write it to disk somehow. You can use a fetch request with batching and regular autorelease pool draining to make sure you don't use up too much memory for temporary objects. Store the data into the directory you get with NSCachesDirectory. You'll obviously want to store the data in a way that lets you relate it back to the object's managedObjectID.

Then, you shut everything down and ask Core Data to migrate the store from version 1 to version 2. See this link for details. Open up the store with version 2.

You might have to add a step where you assign some sort of unique ID to each object, because I'm not sure if Core Data maintains object IDs when it does a non-lightweight migration. If you need to do this, your version 2 model would add a new attribute to the object you're taking the binary data out of that would be either optional or have a default value set. Since lightweight migration shouldn't change the managedObjectIDs, you could at save the mapping of your new unique ID to the managedObjectIDs you saved along with the binary data two paragraphs ago.

Save the data and close the store.

Open the store and do a migration from version 2 to version 3, which should basically be the code you already had written before you posted the question. Once the store is open, add all of the objects you saved from the version 1 store and set up the relationships using the data you saved along the way.

Simple, right?

Outlaw

Thanks Jacques, actually this gave me a good hint and I have done something similar without going as low as SQLite. But I'm creating myself two different stores and creating objects in each of them. The tricky part is to make sure Core Data turns objects into fault as soon as used. – Veinstone