Unit Tests fail, then don't

Question

After some serious frustration and headaches, I've managed to add unit tests to an existing project. The problem is, the tests only get executed 50% of the time. I have a test method with an STFail macro inside, when I run the tests, it fails as I'd expect. Then, I run the tests again without touching any code/settings and it will pass without hitting breakpoints in the test fixture. The console prints out this output when this happens:

The executable for the test bundle at ... octest could not be found.

I've had a Google but there doesn't seem to be many/any people with this exact issue and other discussions on the subject are going over my head. If anyone can shed some light on this issue, It'll be very much appreciated.

Thanks.

Cortes

I don't think this problem will be able to be solved without a reproducible example. Can you remove extraneous code from your unit test, get it so that the same issue is happening, and post that code? – Rolfrolfe

Hi David. There's not a lot to post really. At this stage, it's an empty subclass of a SenTestCase class with one method and a single STFail macro. I haven't even started hooking it up to the application code yet. I can try and remove the application code and upload the project file (if that's possible on here) if that's any use? – Cortes

If it's small enough you could (and should) put it in the post (indenting each line by four characters using the {} button). But that's if you can get it down to something like <100 lines. That way, if it's failing because some setting is wrong, people will be able to see that and tell you – Rolfrolfe

A copy of the code alone isn't enough, because it's probably a problem with the project settings. Perhaps you could strip the project and share it somewhere so we can have a look. – Conakry

Ok, I've reproduced the issue on a fresh project with just a Cocos2d+Box2D template and one test suite. The test fails every other time. You should be able to download it here: dl.dropbox.com/u/26535561/TestApp.zip – Cortes

Answer 1

I took your fresh project and reproduced the problem. There are a number of issues, and one unknown:

Build Phases of test target:

• Target Dependencies: Specify TestApp, so that the main target is always built first.
• Compile Sources: Don't include any main sources. Only include TestAppUnitTests.mm
• Link Binary With Libraries: The only one needed is SenTestingKit.framework. We can throw in Foundation and UIKit for good measure.
• Copy Bundle Resources: Should have nothing. (Later, you may want test-specific resources.)

Build Settings of test target:

• Bundle Loader should be $(BUILT_PRODUCTS_DIR)/TestApp.app/TestApp
• Test Host should be $(BUNDLE_LOADER)

Scheme of main target:

• Test -> Info should specify the test target

But even if I make all these changes, your project still has the odd behavior. The simulator sometimes gets a version of the app named, literally, ${PRODUCT_NAME}. Even debugging the app sometimes shows "Debug ${EXECUTABLE_NAME}" in the Log Navigator. So something else is awry, and I don't know what.

The best solution is to create a new project from scratch, specifying "Include Unit Tests". I've verified that this works.

Conakry

Ok, thanks for the pointers Jon. It sounds like it may be easier to create a new project like you suggested. Thanks for your help anyway. – Cortes

Cocoapods messed up when I refactor target names in my Xcode project. This helped me to fix my test target. Thanks! – Pyromancy

Regarding the bundle settings AND the test host you should see: build/debug-iphoneos/AppName.app/AppName that's a valid path. – Hahnert