SSL Connections Issues when running Unit Tests from the command line

Question

Goal

Our goal is to execute our Unit Tests within a Continuous Integration environment (Jenkins)

(I believe it is essential for every question to state what exactly one is trying to achieve. Maybe the problem can actually be solved a very different way)

Update: More on "Why do you want to do this?"

Firstly, we are talking to specific, self-developed hardware. And I would like the Continuous Integration Test to tell me if anyone changed the behavior of the box without telling all the developers (Yes, yes, I know, such things never happen …)

Secondly, some (not all) of our connections use certificates which are not valid by default, so we have code to check the validity of the certificate (SecTrustEvaluate etc.). Ideally of course, our tests would test that code, too. But that seems too much to ask.

Thirdly: Well, why should I mock anything if I can have the real deal? The IDE has no issue with that, why should the command line.

And if I wanted predictability, I'd return YES in all my tests ;) (and I have seen people do that). No, I want to know if the code actually and really works with our device. Is that some sort of perverted desire?

Problem resolved so far

Running tests from the command line sounds pretty straighforward, but

xcodebuild -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO clean test

results in an ugly error:

unsupported build action 'test'

So I searched the web and found this article on Running OCUnit Tests from Command Line.

I followed all the steps, and I can run my tests from the command line like this:

xcodebuild -scheme CITests -sdk iphonesimulator TEST_AFTER_BUILD=YES ONLY_ACTIVE_ARCH=NO clean build

Remaining Problem

However, now any NSURLConnection to an SSL Server will fail, because "The certificate for this server is invalid.". I have heared about keychain issues when running tests from the command line, but can this be true? Any SSL connection is refused?

To Reproduce

Get the sample at https://github.com/below/SSLTestDemo. Open it up, and run the sample test using Xcode's own Test ⌘U command. The test should succeed.

Now run the test on the command line:

xcodebuild -scheme CITests -sdk iphonesimulator TEST_AFTER_BUILD=YES ONLY_ACTIVE_ARCH=NO clean build

The test fails due to the "The certificate for this server is invalid." error.

Any Pointers?

• Am I doing it wrong?
• Is this a genuine bug?
• If so, are there workaroungs?

Any input is appreciated!

Meaningless

Mh why do you unit test with network connection? Normally you mock such things for more predictable outcome and eliminate the external dependencies. – Inseminate

See my addendum above – Meaningless

to answer your question, "why mock when you can have the real thing": You want to eliminate variables. If you actually connect to the device in the unit test a test failure could result from any number of things other than a code commit. Such as the device is powered of, not connected to the network. Not to say these tests have no value, but they're not unit tests. – Bill

Right. These are integration tests, which is what we are trying to do. – Meaningless

When running as a unit test you can run into an SSL host checking problem. See quellish.tumblr.com/post/33284931593/… – Extenuatory

Answer 1

Well, I ended up simply not checking certificates when we are testing under Jenkins.

As it is dangerous to have code which prevents such checking in your app — after all, it just might not get removed before you ship — you have to set both an environment variable and a compiler flag to activate it …

Post Scriptum:

Apple considers this a bug, rdar://problem/10406441

Meaningless

Answer 2

I'm having the SSL connection issues even within XCode (with XCode 4.5)

My solution was to switch of HTTPS Certificate Checking within the setUp of my Unit Tests:

- (void)setUp
{
    // Set-up code here.
    [super setUp];

    NSURL *URL = [NSURL URLWithString:<#Your SSL Address#>];

    [NSURLRequest.class performSelector:NSSelectorFromString(@"setAllowsAnyHTTPSCertificate:forHost:")
                             withObject:NSNull.null  // Just need to pass non-nil here to appear as a BOOL YES, using the NSNull.null singleton is pretty safe
                             withObject:[URL host]];
}

The esoteric format of the call I make is because this is a private method which refuses to compile with the latest toolchain. As this is only used in the Unit Test target it will not affect the production code and is therefore safe from attack, mistake and Apple AppStore approval. ymmv.

Zeeland

You can use a class extension to allow access to that method from inside your test. See my previous comment for an example. – Extenuatory