OCUnit tests fail from the command line but work in Xcode when using Keychain Services

Question

I'm using SFHFKeychainUtils to use Keychain Services in my app. I've written some OCUnit tests that verify the funcionality of this code. Everything works fine when I run the unit tests from Xcode on the iOS simulator or my device. However now I'm trying to set up a CI server and the test is failing when it is run via the command line with error code -25291. Looking that up on Apple's documentation tells me: No trust results are available (errSecNotAvailable). I've linked the Security.framework to my unit test project, it seems like from what I have read on the web this is all I should need to get this working. Here is the command I am invoking in the console:

/usr/bin/xcodebuild -target [Test_Target] -sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.1.sdk/ -configuration Debug

Does anyone have any experience or suggestions for getting unit testing and Keychain Services to play nicely together from the command line?

Feck

Are you running the CI server in headless mode, or do you have an active user session? – Interglacial

I have an active user session. – Feck

Very odd, can you confirm you have access to the keychain, i.e. it isn't password protected? – Interglacial

we have an intermittent issue with this on Jenkins CI. Have you made any progress? – Gorcock

Sorry it has taken me so long to get back to you guys on this. @Stuart Ervine - Yes, confirm keychain access. In the end the only solution I have found was to have the simulator running like quellish has suggested. The only problem with this is that it seems like a bit of a hack, and is not quite as reliable imo. – Feck

I've successfully set up TeamCity to build and run unit tests and automation tests (a massive pain!). I've put the basics of how I scripted it in github. Checkout: github.com/stuartervine/xcode-sh/blob/master/build.sh - it may need a little work as I haven't touched it for a while. The script creates a keychain and imports your certifications / keys, then uses this for the default keychain for the xcodebuild task. I hope this is of some use... – Interglacial

Answer 1

I ran into the same issue, and the solution for me was to make sure the simulator was running before starting any test. I did that using AppleScript in a Run Script build phase in Xcode, and essentially the same thing on the CI server. Here is the shell script that will open the simulator:

exec osascript <<EOF

tell application "iOS Simulator"

activate

end tell

The security/keychain services issue that causes this is apparently a known issue, though I don't yet have the radar that tracks it. If you're using Jenkins, put the above script in a Execute Shell phase before your Xcode build phase. If you're controlling this through Xcode itself, put it in a Run Script build phase before the RunUnitTests Run Script build phase. Hope that solves your issue!

Vibraculum

THANK YOU. There went an hour of my time prior to Googling the issue. The script worked exactly as you had suggested. – Teide

Quellish - you might want to checkout the WaxSim project. This allows you launch the simulator from the command line. github.com/square/WaxSim - Also you can use the following to kill any active sims ' ps aux | grep -e 'iPhone Simulator.app' | grep -v grep | awk '{print $2}' | xargs kill -9' – Interglacial

@stuart... instead of the complex grep and aux solution... you should be able to use "killall -9 'iPhone Simulator'" – Ithyphallic

Doesn't work for me 34:45: execution error: Can’t get application "iPhone Simulator". (-1728) – Jacqui

Answer 2

I wasn't able to figure out why keychain access fails when OCUnit tests are run from the command line.

In order to proceed with my testing I added a hacktastic category to my unit test target:

//
//  SFHFKeychainUtils+UnitTestHacks.m
//  CB30
// 
// GRRR!!! https://mcmap.net/q/372596/-ocunit-tests-fail-from-the-command-line-but-work-in-xcode-when-using-keychain-services
//
//  Created by Joshua Vickery on 5/14/12.
//

#import "SFHFKeychainUtils+UnitTestHacks.h"

static NSMutableDictionary *fakeKeyChainHolder;

@implementation SFHFKeychainUtils (UnitTestHacks)

+ (NSMutableDictionary *)fakeKeyChainForServiceName:(NSString *)serviceName {
    if (nil == fakeKeyChainHolder) {
        fakeKeyChainHolder = [NSMutableDictionary dictionary];
    }

    NSMutableDictionary *fakeKeyChain = [fakeKeyChainHolder objectForKey:serviceName];
    if (nil == fakeKeyChain) {
        fakeKeyChain = [NSMutableDictionary dictionary];
        [fakeKeyChainHolder setObject:fakeKeyChain forKey:serviceName];
    }
    return fakeKeyChain;
}

#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wobjc-protocol-method-implementation"

+ (BOOL) deleteItemForUsername: (NSString *) username andServiceName: (NSString *) serviceName error: (NSError **) error 
{
    [[self fakeKeyChainForServiceName:serviceName] removeObjectForKey:username];
    return YES;
}


+ (BOOL) storeUsername: (NSString *) username andPassword: (NSString *) password forServiceName: (NSString *) serviceName updateExisting: (BOOL) updateExisting error: (NSError **) error 
{
    [[self fakeKeyChainForServiceName:serviceName] setObject:password forKey:username];
    return YES;
}

+ (NSString *) getPasswordForUsername: (NSString *) username andServiceName: (NSString *) serviceName error: (NSError **) error 
{
    return [[self fakeKeyChainForServiceName:serviceName] objectForKey:username];
}

#pragma clang diagnostic pop

@end

Please note that this is not a good solution but a work-around to get unit tests working until a better solution is found.

Dineen

I still think in a way this IS a good solution because you're effectively mocking out the framework, never a bad thing in a unit test.. – Teide

Answer 3

I am experiencing a similar problem. From my research this might come down to the simulator version that's running when the tests are running. Unit testing keychain with iphone simulator 6.0 up will always make my tests fail, if running from the command line. Change it to any other version (4.3, 5.0, 5.1) and they pass. Always good from XCode with any version though.

Maybe this is a command line tools problem which is not setting up some necessary flags before running tests.

I opened up a new question here with a small test case: Keychain Services API fails with errSecNotAvailable in iphonesimulator 6.0

Madder