OCUnit tests failing, but Xcode Log Navigator says No Issues

Question

Im having a very sneaky issue in Xcode and its "Log Navigator".

I've recreated the issue like so; I have a very simple OCUnit test:

-(void)testSimpleFailure{

    STFail(@"Please fail!");
}

I clean my build, I build my test scheme, and run my tests.

In the output console window I see all my test execute, including the one above. The console displays the test actually ran AND failed:

##teamcity[testFailed name='-|[SimpleTestClassTests.testSimpleFailure|]' message='Please fail!' details='']

HOWEVER, Xcode displays the "Tests Succeeded overlay" and the "Log Navigator" on the left says there are no issues and all is green.

Has anyone else had a similar issue? Does anyone know how to resolve this?

I would very much like to not get a false positive before I commit.

Edited to clarify my question.

Fluidextract

Answer 1

I'm running my application-tests and unit-tests on a device. I'm getting the same results as you - tests are failing but the Log Navigator says everything passes. Not only that, I can't set breakpoints in my code.

The solution for me was to "Edit Scheme...", chose the "Test" scheme, and change my Debugger from "LLDB" to "GDB". I'm using Xcode 4.3.2, with my compiler set for all targets to LLVM 3.1

After changing my debugger for the Test scheme to GDB, The Log Navigator reports the test failures, and I can set breakpoints and debug as well while the tests are running.

link: search for "Unit tests crash before executing any code" (LLDB bug) https://github.com/kstenerud/iOS-Universal-Framework

Iila

If I could give you more upvotes, I would. I've spent all day banging my head against this. It worked on my simple test case. I am hoping it does the trick on the actual project tomorrow. – Waterfowl

Same here. Switching to GDB fixed it. – Postmeridian

Answer 2

Ok, after about 3 developers over 5 days looking at this problem, we have found a solution:

rm -rf Library/Application\ Support/iPhone\ Simulator

Or, alternatively, if blowing away an entire directory tree isn't your thing:

Go to your iOS Simulator application. Choose iOS Simulator from the top menu and choose "Reset Content and Settings..."

Apparently something gets corrupted between the simulator and Xcode, so Xcode cannot hook to the simulator to watch the tests. Xcode believes no tests ran, therefore the tests pass.

Exactly what causes this, we aren't sure, but after 5 grueling days and 1 solution later, we don't care because we're heading to the nearest bar.

Cheers.

Fluidextract

Have you seen this behavior when unit testing on device? My "solution" was to set a break point at the top of the test I'd just written and step through. Did you file a radar? I'd like to dupe it. – Cookie

Answer 3

I just read this long post about the issue that also affects Mac apps:

http://masonmark.com/the-xcode-fairy/

The trick seems to be to quit Xcode and open the project again. The bug is intermittent and will sometimes be here, or never be here, and it stays in this state until the next restart of Xcode.

It's a good read too.

Sectional

Answer 4

Yes, I've seen this, intermittent and annoying.

Do make sure the top (most recent) log message is selected.

Prime

Luckily, each of the Log Navigator logs has a time stamp, so I'm sure I've selected the correct log. – Fluidextract

Yeah, I understand and I know that the error you mention is real, I get it too. It can be rather embarrassing (and to Apple in abstentia ) when I have that problem during demonstrating TDD. :-) – Prime

How do you get around this? This has been happening for the last day. – Fluidextract

Answer 5

What worked for me: Reset the Simulator like mentioned by Nick L in his answer PLUS closing XCode and restarting. Then every build with an STAssertTrue(false) or STFail will give me a negative result.

Mordy

Answer 6

If you are using XCTest, make sure you are deploying to an iOS 7 simulator. If you deploy to iOS 6 it will say build succeeds, test fails but not give any logging information in the debugger.

Il