Core Data "The Database appears corrupt" -- What causes this error?

Question

I'm banging my head against the wall here, I'm using Core Data for a SQLLite DB, and I am able to successfully save to the database (I've checked the contents in an offline SQLLite browser), but after saving the first query I attempt to run comes back with the error seen below, and I can't find any helpful information on the internet relating to this specific error:

Core Data: error: -executeRequest: encountered exception = The database appears corrupt. (invalid primary key) with userInfo = { NSFilePath = "/Users/user/Library/Application Support/iPhone Simulator/7.0.3/Documents/db.sqlite";

The question here, is what causes this error, as I can't find any information relating to it.

For a little background, this is my setup, please assume I've got good reasons for the design that has been made and don't provide an answer of "Change your design" unless you can see something fundamentally broken with the pattern itself.

I have 3 Managed object contexts, all of them are NSPrivateQueueConcurrencyType, the first (A) is attached to the Persistent Store Coordinator, the second (B) has A set as its parent context, and the third (C) has B set as it's parent context -- A chain. The reason for this is that C is a writable context, fetching data from a network source and synchronizing it and saving it, B is a context shared by the UI elements, and I want it to be responsive, finally A is a background context designed to offload any delays for saving to disk off Context B & C

PSC <-- A <-- B <-- C

If I take out the last step (Saving A to PSC) then the application runs great, keeping everything in memory and querying against the in memory contexts. The crash only occurs after I add the save step back in, and only on the first query run against the DB after that save. Both my Save's and my fetch execution are wrapped in performBlock:

Here is the last save:

- (void)deepSave
{
    // Save to the Save Context which happens in memory, so the actual write to disk operation occurs on background thread
    // Expects to be called with performBlock

    NSError *error = nil;
    [super save:&error];
    NSAssert(!error, error.localizedDescription);

    // Trigger the save context to save to disk, operation will be queued and free up read only context
    NSManagedObjectContext *saveContext = self.parentContext;
    [saveContext performBlock:^{
        NSError *error = nil;
        [saveContext save:&error];
        NSAssert(!error, error.localizedDescription);
    }];
}

And here is the execute stack (on NSManagedObjectContext Queue thread)

#0  0x0079588a in objc_exception_throw ()
#1  0x079d98e7 in -[NSSQLiteConnection handleCorruptedDB:] ()
#2  0x078d9b8d in -[NSSQLiteConnection fetchResultSet:usingFetchPlan:] ()
#3  0x078e24a5 in newFetchedRowsForFetchPlan_MT ()
#4  0x078cd48e in -[NSSQLCore newRowsForFetchPlan:] ()
#5  0x078cca8d in -[NSSQLCore objectsForFetchRequest:inContext:] ()
#6  0x078cc53f in -[NSSQLCore executeRequest:withContext:error:] ()
#7  0x078cbf62 in -[NSPersistentStoreCoordinator executeRequest:withContext:error:] ()
#8  0x078c96c6 in -[NSManagedObjectContext executeFetchRequest:error:] ()
#9  0x0791e526 in -[NSManagedObjectContext(_NestedContextSupport) _parentObjectsForFetchRequest:inContext:error:] ()
#10 0x0799c1f4 in __82-[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:]_block_invoke ()
#11 0x0791e321 in internalBlockToNSManagedObjectContextPerform ()
#12 0x013c34b0 in _dispatch_client_callout ()
#13 0x013b0778 in _dispatch_barrier_sync_f_invoke ()
#14 0x013b0422 in dispatch_barrier_sync_f ()
#15 0x0791e2a2 in _perform ()
#16 0x0791e14e in -[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:] ()
#17 0x078c96c6 in -[NSManagedObjectContext executeFetchRequest:error:] ()
#18 0x0791e526 in -[NSManagedObjectContext(_NestedContextSupport) _parentObjectsForFetchRequest:inContext:error:] ()
#19 0x0799c1f4 in __82-[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:]_block_invoke ()
#20 0x0791e321 in internalBlockToNSManagedObjectContextPerform ()
#21 0x013c34b0 in _dispatch_client_callout ()
#22 0x013b0778 in _dispatch_barrier_sync_f_invoke ()
#23 0x013b0422 in dispatch_barrier_sync_f ()
#24 0x0791e2a2 in _perform ()
#25 0x0791e14e in -[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:] ()
#26 0x078c96c6 in -[NSManagedObjectContext executeFetchRequest:error:] ()

Stinkwood

The only time this happens is if one of the rows in your database has z_pk=0. Identifying that row and determining where it gets created would be the first step in debugging this. – Ritch

@Ritch -- I've opened the DB in a SQLLite browser, and at the time of crash there is only one entity table that has data in it (the entity that I am running a query against) and it has normal, sequential Z_PK values starting with 1 :( – Stinkwood

In that case threading could be a culprit, but appropriate use of performBlock should solve that. At this point the rows are being read directly from the SQLite API, so there's little chance that CoreData is lying when it says the DB is corrupt. If you're using WAL mode, try turning that off (switch to delete journal mode) and repeat your tests. If it fails you should be able to find the offending row. – Ritch

@Ritch Could you take a look at the solution I found? https://mcmap.net/q/846626/-core-data-quot-the-database-appears-corrupt-quot-what-causes-this-error – Stinkwood

There is a fundamental problem, child contexts are not designed for threading. – Hess

Hi can you tell or share, how you setup PSC <-- A <-- B <-- C – Handmade

Answer 1

Okay, I've tracked it down. It appears that there is something broken in propertiesToFetch against NSManagedObject resultType (not supposed to be used, our mistake) in this setup -- against a context that has a parent context instead of a persistent coordinator. This unit test shows that all you have to do is set a property to fetch to get this error (while doing the query without the properties to fetch works correctly). The fix here for us was to stop incorrectly using properties to fetch :)

- (void)testManagedObjectContextDefect
{        
    NSManagedObjectContext *contextA = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
    contextA.persistentStoreCoordinator = sqllitePersistentStoreCoordinator;
    NSManagedObjectContext *contextB = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
    contextB.parentContext = contextA;

    NSEntityDescription *entityDescription = [NSEntityDescription entityForName:@"GCSCObject" inManagedObjectContext:contextB];

    [contextB performBlockAndWait:^{
        GCSCObject *object = [[GCSCObject alloc] initWithEntity:entityDescription insertIntoManagedObjectContext:contextB];
        object.serverID = @"1";
        NSError *error = nil;
        XCTAssert([contextB save:&error] && !error, @"Failed to save - %@",error); // B -> A save
    }];

    [contextA performBlock:^{
        NSError *error = nil;
        XCTAssert([contextA save:&error] && !error, @"Failed to save - %@",error); // A -> PSC, background save
    }];

    [contextB performBlockAndWait:^{
        NSFetchRequest *request = [[NSFetchRequest alloc] initWithEntityName:@"GCSCObject"];
        NSError *error = nil;
        NSArray *results = [contextB executeFetchRequest:request error:&error];
        XCTAssert(results.count == 1 && !error, @"Fetch failed to retrieve - %@ / %@",results,error);
        GCSCObject *object = results[0];
        XCTAssert([object.serverID isEqualToString:@"1"], @"Value retrieval failed");

        // Everything passes up to here, so far so good!

        request = [[NSFetchRequest alloc] initWithEntityName:@"GCSCObject"];
        request.propertiesToFetch = @[@"serverID"]; // This is the culprit of the index crash
        results = [contextB executeFetchRequest:request error:&error];
        XCTAssert(!error, @"%@", error.localizedDescription); // !!! HERE we have a failure, assert: "Core Data: error: -executeRequest: encountered exception = The database appears corrupt.  (invalid primary key) with userInfo = { NSFilePath = "/path/db.sqlite }";
    }];
}

In this case GCSCObject is a regular entity, and serverID is one of it's parameters (it doesn't matter which parameter is used, or what type it is, I tried with multiple. Here is the description of the serverID parameter I used for this test:

The crash occurs wether or not we provide the andWait to contextA save (though doing so would kind of void the point of having a background queue for saving)

I'd love feedback regarding why this might be the case, but for now, not using properties to fetch allows our application to work smoothly. I'm contemplating filing an Apple Bug here.

Stinkwood

You can't use propertiesToFetch without requesting a NSDictionaryResultType so I am not sure how this works in any situation. Also, you are supposed to pass in NSPropertyDescription instances to propertiesToFetch not instances of NSString. However, under normal circumstances neither of those problems would cause you an error. However, as a next step I would suggest removing those and confirming the issue still exists without them. As an aside, I would not file an Apple Radar with the code in its current state, Apple will tell you the same things I am telling you up to this point. – Presto

Are your referring to the Unit Test (above) -- I think it's pretty clear. It would fail for you as well I think, The last step runs the query without the fetched properties, and then with, and it only fails in the with scenario (of note, it doesn't do so if you do the same thing against a PSC directly)? If you are curious, I'd like to share the code we devised to allow for using NSFetchedResultController on a private queue - now that we found the bug in propertiesToFetch, it's working swimmingly again, fast no lag table scrolling on large data sets, only nominal delay before cells are drawn. – Stinkwood

I don't think you are following me. You are stating that the propertiesToFetch is causing a crash. I am telling you that you are using it wrong. In most cases your use of it is ignored (tested). In your specific example you have found a way to make it crash. This is most likely not an Apple error but a developer error. If you change the result type to dictionary does it still crash? – Presto

I would still file the bug with Apple, but for a better error. API abuse of this type should result in something easier to debug than a corrupt database. – Ritch

Thank you for your answer. This saved me hours of digging! – Gorges

propertiesToFetch: " If NSManagedObjectResultType is set, then NSExpressionDescription cannot be used, and the results are managed object faults partially pre-populated with the named properties" so it would appear you CAN use it. – Hess

I faced this very issue when trying to use propertiesToFetch while fetching managed objects (hence NSManagedObjectResultType). In my case it's clear: if I set propertiesToFetch to some (valid value), no matter whether it's array of strings (keys) or NSPropertyDescriptions (corresponding to that very keys), I get that "The database appears corrupt. (invalid primary key)" error. So it looks like even though the documentation appears to say that it should work, in practice it doesn't. Just in case, I get it on iOS 10.1. – Lefthand

Looking at the most recent docs for propertiesToFetch: This property *can* be set with *managedObjectResultType* and thereby implement a partial faulting (whereby only some of the properties are populated) of the returned objects *as well as* the *dictionaryResultType*. Even still, I get a crash with a parent managed object context. – Subtropics

Answer 2

If you experience this error when performing a fetch request that is retrieving some aggregate result (sum, max, min, ...) make sure you set

fetchRequest.resultType = NSDictionaryResultType;

Charmion

This was the problem for me. Specifically, I was using the BNRCoreDataStack because I'm too lazy to set up my own knowing how it all works anyway. I was building an NSFetchRequest for an NSFetchedResultsController where I set the request's propertiesToFetch but not the resultType to NSDictionaryResultType. github.com/RestKit/RestKit/issues/1642#issuecomment-39413788 – Giselle

Answer 3

First, your UI context should not be a private queue context. That is what the NSMainQueueConcurrencyType is for.

Second, do not check the error on save. Check the BOOL return from the -save:. That error can have junk in it even on a successful save.

Third, what do your other two saves look like? If they are all on private queues and being saved async then you may be running into a race condition. C should save synchronously, B should save synchronously and then A should be async.

Presto

UI Context is a private context because we're trying to keep the main thread as responsive as possible. Our table views, etc are correctly using the performBlock -> do a query -> copy result strings out of NSManagedObject -> Dispatch async to main thread with changes. More of a pain but keeps the heavy fetch queries off the main thread. I'm looking into your other suggestions, thanks! – Stinkwood

In regards to your second point, I've changed the code to catch that but sadly got no new errors :( Finally, in regards to your third point, here is what the other two saves are like -- pastie.org/8809824 I am keeping the C->B and B->A saves in a synchronous call :/ – Stinkwood

Setting the UI context to private is not going to give you a performance benefit. It will slow things down. Either your fetches for UI display are going to be async or you are going to be blocking for them. Worse, things like the NSFetchedResultsController expect to have a main thread NSManagedObjectContext. I doubt using a private context like you are is inherently stable. – Presto

Do not check the error in your asserts. You should ONLY check the BOOL. The error is meaningless unless that BOOL returns NO. – Presto

We need to do the fetches for UI display Async :) There is a slight bit of delay added because you are putting the work of display off until the run loop after the query is done (as opposed to the same run loop), however if you block for the query then touches, slides, etc are all backlogged behind the query itself (UI lag). Fetched Result controller delegate responses can be dispatched to the main thread to work on the table (which we are doing). The situation which this is failing doesn't involve the UI context (B) at all, as it is failing during a unit test that saves C and then Queries C – Stinkwood

This still smells like a threading issue and the way you are handling the main thread is violating the thread containment rules of Core Data. The NSFetchedResultsController expects to be using a main thread context and when you are using a private context it might "work" but it violates the containment and is going to cause problems. The fact that you are having problems seems to be more than coincidence. – Presto

The unit test doesn't use context B at all. No Fetched Result controllers, etc. Two acts are all that is required to cause the crash: 1. Make some changes to context C and Save down through B->A->PSC, 2. Query against Context C. – Stinkwood

Marcus -- Could you take a look at the solution I found? I'm able to easily reproduce this crash now using a very simplified stack... https://mcmap.net/q/846626/-core-data-quot-the-database-appears-corrupt-quot-what-causes-this-error Thanks for your feedback! – Stinkwood