how to write expected failures?

Question

In Xcode, at the end of my unit tests I get a result like this:

Test Suite 'All tests' finished at 2012-12-06 10:23:38 +0000

Executed 195 tests, with 0 failures (0 unexpected) in 4.314 (4.485) seconds

I would love to find out how can I define tests with expected failures.

Normally with other test frameworks I like being able to just define incomplete unit tests as reminders of future work that must be done. These tests should be logged only as warnings, but still yield a "Success" final result if everything else is OK

Looking at the output of Xcode, I assume there is a way to achieve the same. However I am having problems finding the right macro to mark incomplete/TODO tests. Furthermore it seems strange to me that normal failures are reported as:

Executed 95 tests, with 1 failure (0 unexpected) in 2.314 (2.334) seconds

Hence, any test assertion failure seem to be expected. In that case I am even confused with the meaning of (0 unexpected) failures.

Could anyone explain the meaning of that part of the log results?, how can it be used? how can I mark incomplete tests?

Ori

Answer 1

An "unexpected failure" is a thrown and unhandled exception. As opposed to a possibly expected failure, which is checked by an assertion.

OCUnit does not provide a way to mark tests as "do not run." Either comment them out (which leaves the risk of the test no longer compiling) or change the method name so it doesn't have a "test" prefix. For example, rename testSomething to XXXtestSomething. (This leaves the risk that you'll forget to change it back.)

Marybethmaryellen

Thanks! once time more again, you are the great cocoa testing hero. My question came up after after a full 100% BDD in Ruby with rspec + cucumber adventure. Now I understand every rant about the TDD landscape in Cocoa projects. – Ori

Would love if this was documented somewhere. But Apple's unit testing leaves a lot to be desired. – Amytal

Answer 2

One solution to my problem is using the warning preprocessor macro to mark unit tests that must be completed. That way after compiling unit tests I get a good overview of future work that must be done. This is shown in the warnings panel, hence it is much more attractive than just using TODO comments.

However I would rather like to have a total count of incomplete tests and finding out the meaning of the always "0 unexpected" count of my log reports.

Ori