What is the purpose of XCTestCase's setUp method?

Question

Per the comment within the default template for XCTestCase regarding setUp :

Put setup code here; it will be run once, before the first test case.

However, in XCTestCase.h, the comment above setUp states differently:

Setup method called before the invocation of each test method in the class.

To confirm the actual behavior, I put an NSLog withinsetUp to count how many times it was called:

static int count = 0;

- (void)setUp
{
    [super setUp];
    count++;

    NSLog(@"Call Count = %d", count);
}

This resulted in the setUp method being called before every test method (confirming the comment on XCTestCase.h).

I wanted to use the setUp method to create test/mock objects once (e.g. to setup a Core Data test stack). Creating these over and over again would be processor intensive and potentially very slow.

So,

1) What is setUp actually intended to be used for? Surely developers aren't creating objects in it over and over?

2) How can I create these objects only once within an XCTestCase?

Scalpel

Answer 1

There are a couple of points for discussion here: the behaviour of the setUp methods, and general best testing practices.

There are actually two setUp methods:

+ (void)setUp;
- (void)setUp;

The class method (+ (void)setUp) is only run once during the entire test run.

The instance method (- (void)setUp) is the one in the default template; it's run before every single test. Hopefully, in a hypothetical future version of Xcode, this comment will have been changed to // Put setup code here. This method is called before the invocation of each test method in the class. WINK WINK

So through these two methods, both of the behaviours you described are possible.

Regarding your comment:

"Surely developers aren't creating objects in it over and over?"

My answer would be "Yes, they usually are". A popular acronym for 'good' unit tests is FIRST:

• Fast
• Isolated
• Repeatable
• Self-Verifying
• Timely

Isolated is key to this discussion: your tests shouldn't rely on any previous state left behind by other tests. Ideally, you should tear down and recreate your in-memory Core Data stack for every test, so you know that you're starting from a clean slate. A good example is in this post by Graham Lee. You want to use an in-memory stack because a) you can easily throw it away, and b) it should be very fast because it's just in-memory and not hitting your disk.

If you find that your tests are running slowly as a result of this (don't optimize prematurely), then I think a reasonable next step would be to create the stack in your + (void)setUp method, but create a brand new context every time in your - (void)setUp method.

Stodgy

+1: Wonderful answer! Yes, thinking about it, 1) test order isn't guaranteed (yet with good tests, it shouldn't matter which happens when) and 2) tests shouldn't be affected by what other tests are doing (as you mention, they shouldn't rely on any previous test state). Good idea with the in-memory stack too. This is just what I will do. – Scalpel

After reading this answer, I immediately deleted mine +1 ;) Just a note: XCTest is more or less a copy of SenTest inheriting all its limitations. There are much better testing frameworks which have groups which can be arbitrarily nested, where the setup and teardown procedure is obvious. Unfortunately, Xcode and XCTest is far behind, and integration of really cool third party testing frameworks into the exciting testing frame of Xcode is quite difficult. – Sternutatory

@Scalpel Exactly right - Test order isn't guaranteed. @Sternutatory For instance, specta has beforeEach, beforeAll, afterEach, and afterAll methods. – Stodgy

Just a comment on making use of the class method instead of the instance method: I have a plist file which describes a large data structure and a class which parses this file at runtime to generate a collection of objects. These objects are useful independently of each other so instead of writing a method to expose the parsing of single items from the plist, or specifically exposing the method which handles parsing, I parse then entire structure once for the Test Case and reuse its objects in each test. Reparsing this file repeated wouldn't make sense for testing. – Titulary

Just noting that the future is now - Xcode's template for tests now includes // Put setup code here. This method is called before the invocation of each test method in the class. verbatim. Was that an edit, or did you predict the future? :) – Wittie

This answer makes a good point, but completely misses the possibility that some values may remain constant for the whole test. Then, setting them up once for all tests does not violate the cited principles (and helps "fast" along). – Granulate

Answer 2

I came here with almost the same problem: how to perform the setUp just once and in Swift. Here is the solution:

override class func setUp() {
    super.setUp()    
    // ...
}

override class func tearDown() {    
    // ...
    super.tearDown()
}

Right now I'm still looking for a solution for an asynchronous setUp!

Gamophyllous

An asynchronous setUp would not seem to be useful, since by definition, you want your setUp to be complete before you run any test in that class. – Trash

I think he means he wants the test functions in his test case not to be called before a asynchronous call in the setUp method finishes. You can use DispatchSemaphore for that. – Cyrilcyrill