Ordering unit test using XCTest in Xcode 6

Question

I would like to know if there is any way to tell Xcode to run unit tests in a specified order. I mean not in a same XCTestCase class file, but between all the class file.

For example I want to run the SitchozrSDKSessionTest before running SitchozrSDKMessageTest.

I looked over few threads on stack or on Apple documentation and I haven't found something helpful.

Thanks for your help.

Conrado

Not that I've found, I believe the theory is that unit tests are atomic and independent so it doesn't matter what order they run in. You can run just one suite of tests by pressing play next to the Test file you want. – Cherisecherish

For exemple these are test for API call. For each call I need to have a token which is retrieve by a specific call. Once I retrieved the token I set it in my HTTP client. I would like to perform only one time the call to retrieve the token and the be able to perform all the other calls. What I've done to "hack" it is all my test classes which are testing API call inherit from the test class which perform the call to retrieve the token. Then before each call to the api I call [super getToken]. It works but I wanted to know if there was other solutions. – Conrado

@Conrado As others have mentioned, every test is supposed to be atomic and independent. Have you tried mocking out the token retrieval? – Imperfective

@Conrado seems like getting the token should be done in your setUp() function. – Microscopium

Answer 1

It's all sorted alphabetically (classes and methods). So when You need some tests running last, just change the name of Class or Method (for (nasty) example by prefixing 'z_').

So...You have Class names:

MyAppTest1
  -testMethodA
  -testMethodB
MyAppTest2
  -testMethodC
  -testMethodD

and they run in this order. If you need to run MyAppTest1 as second, just rename so it's name is alphabetically after MyAppTest2 (z_MyAppTest1) and it will run (same for method):

MyAppTest2
  -a_testMethodD
  -testMethodC
z_MyAppTest1
  -testMethodA
  -testMethodB

Also please take the naming as example :)

Crinkumcrankum

it should be noted that this is not true anymore in Xcode 8 – Entomophagous

Xcode 8.2.1: work. Please make attention: you should rename classes, not files. – Loathe

Answer 2

It is true that currently (as of Xcode 7), XCTestCase methods are run in alphabetical order, so you can force an order for them by naming them cleverly. However, this is an implementation detail and seems like it could change.

A (hopefully) less fragile way to do this is to override +[XCTestCase testInvocations] and return your own NSInvocation objects in the order you want the tests run.

Something like:

+ (NSArray *)testInvocations
{
    NSArray *selectorStrings = @[@"testFirstThing",
                                 @"testSecondThing",
                                 @"testAnotherThing",
                                 @"testLastThing"];

    NSMutableArray *result = [NSMutableArray array];
    for (NSString *selectorString in selectorStrings) {
        SEL selector = NSSelectorFromString(selectorString);
        NSMethodSignature *methodSignature = [self instanceMethodSignatureForSelector:selector];
        NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:methodSignature];
        invocation.selector = selector;
        [result addObject:invocation];
    }
    return result;
}

Of course, the downside here is that you have to manually add each test method to this instead of them being picked up automatically. There are a few ways to improve this situation. If you only need to order some of your test methods, not all of them, in your override of +testInvocations, you could call through to super, filter out those methods that you've manually ordered, then tack the rest on to the end of the array you return. If you need to order all the test methods, you could still get the result of calling through to super and verify that all of the automatically picked up methods are covered by your manually created, ordered result. If not, you could assert, causing a failure if you've forgotten to add any methods.

I'm leaving aside the discussion of whether it's "correct" to write tests that must run in a certain order. I think there are rare scenarios where that makes sense, but others may disagree.

Hypso

That's interesting. Do you know if those methods actually need to be named as "testXXX" ones? If not, probably you don't need to filter anything special - just add to the "[super]" array your methods... P.S. I have following problem: I need to create, process and delete some records. And I want to measure performance of each stage. But if put all those routines in same method, I get "Can only record one set of metrics per test method". So I hope to split those to 3 different methods, still I need to ensure order of execution. – Counterstamp

@NickEntin: No, I don't think they need to be prefixed with "test". That's used by XCTestCase to automatically pick up on methods for its implementation of +testInvocations, but if you're providing the invocations yourself, you should be able to name them however you'd like. I still like the "testXXX" convention, myself. Regarding your problem, you should ask a separate question and link it here. – Hypso

NSInvocation is unavailable in Swift, any workaround? – Ciaracibber

@fede1608: This method has to return an array of NSInvocations. That's just how it works. So, I think the only non-ugly workaround is to implement this in Objective-C. You could presumably make the XCTestCase subclass itself, and this method, be ObjC, then use a Swift extension to add the actual test cases. – Hypso

Using Xcode 8.2.1 and attempting your solution, I'm seeing one of my test methods being called before -testInvocations is called when I run my test suite for the test class. Have you tried this in 8.2.1? – Circumvent

@EvanR: I have not tested this in 8.2.1. If what you're describing is actually happening, that sounds worth filing a radar for. – Hypso

Answer 3

its ordered by function names letter orders, doesn't matter how you order it in your code. e.g.:

-(void)testCFun(){};
-(void)testB2Fun(){};
-(void)testB1Fun(){};

the actual execute order is :

1. testB1Fun called
2. testB2Fun called
3. testCFun called

Agle

I don't want to order test in a specific test class file but but the class files themselves. – Conrado

this is very useful! where did you get this information from? is it documented? – Tiebout

I wish @Conrado 's comment had attention back in '14 – Hummer

Answer 4

In addition to andrew-madsen's answer:
I created a 'smart' testInvocations class method which searches for selectors starting with "test" and sorts them alphabetically.
So you don't have to maintain an NSArray of selector names separately.

#import <objc/runtime.h>

+ (NSArray <NSInvocation *> *)testInvocations
{
    // Get the selectors of this class
    unsigned int mc = 0;
    Method *mlist = class_copyMethodList(self.class, &mc);
    NSMutableArray *selectorNames = [NSMutableArray array];
    for (int i = 0; i < mc; i++) {
        NSString *name = [NSString stringWithFormat:@"%s", sel_getName(method_getName(mlist[i]))];
        if (name.length > 4
            && [[name substringToIndex:4] isEqualToString:@"test"]) {
            [selectorNames addObject:name];
        }
    }

    // Sort them alphabetically
    [selectorNames sortUsingComparator:^NSComparisonResult(NSString * _Nonnull sel1, NSString * _Nonnull sel2) {
        return [sel1 compare:sel2];
    }];

    // Build the NSArray with NSInvocations
    NSMutableArray *result = [NSMutableArray array];
    for (NSString *selectorString in selectorNames) {
        SEL selector = NSSelectorFromString(selectorString);
        NSMethodSignature *methodSignature = [self instanceMethodSignatureForSelector:selector];
        NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:methodSignature];
        invocation.selector = selector;
        [result addObject:invocation];
    }
    return result;
}

Footnote: It's considered bad practice making your unit tests depend on eachother

Exerciser

Isn't this what the standard testInvocations does? i.e sort them alphabetically and run them? – Microscopium

Answer 5

For you exemple you can just rename the tests files on your project like this :

SitchozrSDKSessionTest -> t001_SitchozrSDKSessionTest
SitchozrSDKMessageTest -> t002_SitchozrSDKMessageTest

Xcode treat the files using alphabetic order.

Psalmody

Answer 6

Since Xcode 7, XCTestCase subclasses are alphabetically ordered. If you want to ensure that the test methods themselves are also run in alphabetical order, you must override the testInvocations class method and return the invocations sorted by selector.

@interface MyTestCase : XCTestCase
@end

@implementation MyTestCase

+ (NSArray<NSInvocation *> *) testInvocations
{
    return [[super testInvocations] sortedArrayUsingComparator:^NSComparisonResult(NSInvocation *invocation1, NSInvocation *invocation2) {
        return [NSStringFromSelector(invocation1.selector) compare:NSStringFromSelector(invocation2.selector)];
    }];
}

- (void) test_01
{
}

- (void) test_02
{
}

@end

Icefall

Answer 7

The OP did not mention whether CI is available or whether a command line answer would be sufficient. In those cases, I have enjoyed using xctool [1]. It has syntax to run tests down to just a single test method:

path/to/xctool.sh \
  -workspace YourWorkspace.xcworkspace \
  -scheme YourScheme \
  test -only SomeTestTarget:SomeTestClass/testSomeMethod

I will do this to ensure we test the items we think are easiest first.

1.[] facebook/xctool: A replacement for Apple's xcodebuild that makes it easier to build and test iOS or OSX apps. ; ; https://github.com/facebook/xctool

Criswell