Keychain: Item reported as errSecItemNotFound, but receive errSecDuplicateItem on addition

Question

This issue has been bugging me for a while, and I hope someone has insight as to the cause of this. Essentially, I have a small percentage of users who are unable to save/update items to the keychain. The problematic flow of control is as follows:

1. We check for the existence of the item using SecItemCopyMatching. This returns the error code errSecItemNotFound

2. We then try to add the item via SecItemAdd, but this then returns errSecDuplicateItem.

Because of this, we have some users who are unable to update a subset of keychain items at all, requiring them to restore their device to clear the keychain. This is obviously an unacceptable workaround. It seemed to work for them before, but have now got into this non-updatable cycle.

After researching, I've seen issues regarding the search query used in SecItemCopyMatching not being specific enough, but my code uses a common search query wherever possible.

+ (NSMutableDictionary*)queryForUser:(NSString*)user key:(NSString*)key
{
    if (!key || !user) { return nil; }

    NSString* bundleId = [[NSBundle mainBundle] bundleIdentifier];
    NSString* prefixedKey = [NSString stringWithFormat:@"%@.%@", bundleId, key];

    NSMutableDictionary* query = [NSMutableDictionary dictionary];
    [query addEntriesFromDictionary:@{(__bridge id)kSecClass          : (__bridge id)kSecClassGenericPassword}];
    [query addEntriesFromDictionary:@{(__bridge id)kSecAttrAccount    : user}];
    [query addEntriesFromDictionary:@{(__bridge id)kSecAttrService    : prefixedKey}];
    [query addEntriesFromDictionary:@{(__bridge id)kSecAttrLabel      : prefixedKey}];
    [query addEntriesFromDictionary:@{(__bridge id)kSecAttrAccessible : (__bridge id)kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly}];

    return query;
}

The code to do the updating/adding is as follows (sorry for the verbosity):

// Setup the search query, to return the *attributes* of the found item (for use in SecItemUpdate)
NSMutableDictionary* query = [self queryForUser:username key:key];
[query addEntriesFromDictionary:@{(__bridge id)kSecReturnAttributes : (__bridge id)kCFBooleanTrue}];

// Prep the dictionary we'll use to update/add the new value
NSDictionary* updateValues = @{(__bridge id) kSecValueData : [value dataUsingEncoding:NSUTF8StringEncoding]};

// Copy what we (may) already have
CFDictionaryRef resultData = NULL;
OSStatus status = SecItemCopyMatching((__bridge CFDictionaryRef)query, (CFTypeRef*)&resultData);

// If it already exists, update it
if (status == noErr) {
    // Create a new query with the found attributes
    NSMutableDictionary* updateQuery = [NSMutableDictionary dictionaryWithDictionary:(__bridge NSDictionary*)resultData];
    [updateQuery addEntriesFromDictionary:@{(__bridge id)kSecClass : (__bridge id)kSecClassGenericPassword}];

    // Update the item in the keychain
    status = SecItemUpdate((__bridge CFDictionaryRef)updateQuery, (__bridge CFDictionaryRef)updateValues);

    if (status != noErr) {
        // Update failed, I've not seen this case occur as of yet
    }
}
else {
    // Add the value we want as part of our original search query, and add it to the keychain
    [query addEntriesFromDictionary:updateValues];
    [query removeObjectForKey:(__bridge id)kSecReturnAttributes];
    status = SecItemAdd((__bridge CFDictionaryRef)query, NULL);

    if (status != noErr) {
        // Addition failed, this is where I'm seeing errSecDuplicateItem
    }
}

We tried using SecItemDelete instead of checking/updating, but this also returned errSecItemNotFound with SecItemAdd failing straight after. The delete code is:

+ (BOOL)deleteItemForUser:(NSString *)username withKey:(NSString *)itemKey {
    if (!username || !itemKey) { return NO; }

    NSString * bundleId = [[NSBundle mainBundle] bundleIdentifier];
    NSString * prefixedItemKey = [NSString stringWithFormat:@"%@.%@", bundleId, itemKey];

    NSDictionary *query = [NSDictionary dictionaryWithObjectsAndKeys: (__bridge id)kSecClassGenericPassword, kSecClass,
                           username, kSecAttrAccount,
                           prefixedItemKey, kSecAttrService, nil];

    OSStatus status = SecItemDelete((__bridge CFDictionaryRef) query);

    if (status != noErr) {
        // Failed deletion, returning errSecItemNotFound
    }

    return (status == noErr);
}

Whilst we have defined 2 keychain access groups for the application, affected keychain items do not have an access group assigned as an attribute (which by the documentation, means searching will be done for all access groups). I've yet to see any other error code other than errSecItemNotFound and errSecDuplicateItem.

The fact that only a small set of users get into this condition really confuses me. Are there any other considerations I need to take into account regarding the keychain that could be causing this, regarding multithreading, flushing, background access etc…?

Help much appreciated. I'd rather stick with using the Keychain Services API instead of using a 3rd party library. I'd like to understand the fundamental problem here.

Biddable

I don't see any error in your code that could lead to that behavior. Did you update that implementation in the past ? Users could have some old entries that have never been deleted, even though they updated your app. – Candycecandystriped

I don't believe so. Even then, why would a search query say not found, but an an addition with the same search query fail. Where would a conflict occur there with older entries? – Biddable

Well, i don't know, i had this issue when i was changing access group, but the problem was probably somewhere else. Do you know how to reproduce this bug ? You don't set a kSecMatch item like kSecMatchLimitOne, maybe it could lead to unexpected behavior. Apple stated that a request (usually?, should?) have a search key value pair. developer.apple.com/library/mac/documentation/security/… – Candycecandystriped

We've not been able to reproduce this unfortunately. As for the match attribute, "By default, this function returns only the first match found. To obtain more than one matching item at a time, specify the search key kSecMatchLimit with a value greater than 1" leads me to believe it's not required. – Biddable

Answer 1

The unique key for kSecClassGenericPassword is composed of;

kSecAttrAccount
kSecAttrService

To check for its existence, query the keychain store with only these attributes (including kSecReturnAttributes flag).

Including kSecAttrLabel and kSecAttrAccessible will exclude any existing item with the same unique key, but with different attributes.

Once you have confirmed its (non)existence, add the additional attributes and Add or Update.

Valentinevalentino

Thanks! this really helped. Can I ask how you knew this? I read the documentation backwards and forwards and never saw any reference to this... – Publicspirited

@Publicspirited I also had problems with this. They keys and value documentation is a bit hard to follow at times. Under the attributes and keys section, there are references on which attributes are available on what kinds of keychain items. Items of class kSecClassGenericPassword have this attribute – Roswald

Can I include kSecMatchLimit (set to 1) or kSecAttrAccessGroup (to share with my other apps)? – Chalcocite

Turns out that also kSecAttrSynchronizable set to true is part of the uniqueness of the key. My delete/queries had to include this to match. – Canica

I've had this problem too, and I'll try @simon 's solution. However, I'm still not sure why this is happening to a subset of users, and not all users. – Latrena

Re @shad's question of how this could be known: see the documentation of errSecDuplicateItem which does list out the primary keys for each class: developer.apple.com/documentation/security/errsecduplicateitem – Helsell

I ended up here because I had used Apple's KeychainItem wrapper. Turns out it has a bad bug, which you likely hit as soon as you store 2 things with it: change kSecAttrGeneric (everywhere it appears) to kSecAttrAccount – Helsell