Memory issues migrating large CoreData datastores on iPhone

Question

My app has a potentially very large CoreData datastore underneath it (could easily be upwards of 30MB). I've started noticing memory issues when using the automatic migration (addPersistentStoreWithType:configuration:URL:options:error:) so I started looking into the methods of migrating smaller parts of the store to avoid all the CoreData object buildup that happens when you migrate everything at once.

This is discussed in the official documentation in the "Multiple Passes" section, however it looks like their suggested approach is to divide up your migration by entity type, i.e. make multiple mapping models, each of which migrate a subset of the entity types from the complete data model.

The only problem is - what if one entity type is the majority of your datastore? Going by the Apple-recommended approach, that whole entity type is still going to be done in a single migration and the memory issues will presumably persist.

Are there any techniques available to actually migrate a sub-set of entities of a specific type to guarantee that you will not run out of memory when trying to migrate them all?

Thanks in advance for any help.

EDIT: After doing more digging, I have discovered that the Apple-recommended split of the DB into entity types actually only works for non-related entities (as discussed here), so it's even less likely to solve the problems of a real-world DB than I thought when I originally wrote this post.

I'm starting to think that CoreData migrations that are actually done through NSMigrationManager don't scale at all now and you basically can't have a DB that is bigger than about 20-30MB if you want to be able to migrate it on current generation iOS devices. The only viable approach seems to be to short-circuit all the NSMigrationManager / NSMappingModel stuff completely and write the migration completely custom in code. Seems like a huge oversight on Apple's part if this is actually the case.

Boughten

Answer 1

I was able to get around this in the short-term by leveraging "lightweight" migration, as described in http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/vmLightweight.html#//apple_ref/doc/uid/TP40008426-DontLinkElementID_1.

The trick is to pull the NSMigrationManager subclass that is specifically for SQLite store types when you're manually invoking the migration, as shown in the last code sample on that page.

This isn't a general-purpose fix though since it only works if the schema change in your datastore is simple enough that lightweight migration is possible. Still waiting to hear back from Apple as to what the recommended solution is when you're dealing with a non-trivial mapping.

Boughten

Heard back from Apple and their solution is ... ummm ... do it yourself. So basically, if you have a large DB in your app, do everything in your power to make sure your migrations can be done by an inferred / lightweight model. Bleargh. – Boughten