How to Dynamically add XCTestCase

Question

I'm writing a UI Test for a white label project where each app has a different set of menu items. The test taps on each menu item and takes a screenshot (using fastlane snapshot).

Currently this all happens inside one XCTestCase called testScreenshotAllMenuItems() which looks like this:

func testScreenshotAllMenuItems() {
    // Take a screenshot of the menu
    openTheMenu()
    snapshot("Menu")
    var cells:[XCUIElement] = []

    // Store each menu item for use later
    for i in 0..<app.tables.cells.count {
        cells.append(app.tables.cells.element(boundBy: i))
    }

    // Loop through each menu item
    for menuItem in cells.enumerated() {
        let exists = menuItem.element.waitForExistence(timeout: 5)
        if exists && menuItem.element.isHittable {
            // Only tap on the menu item if it isn't an external link
            let externalLink = menuItem.element.children(matching: .image)["external link"]
            if !externalLink.exists {
                var name = "\(menuItem.offset)"
                let cellText = menuItem.element.children(matching: .staticText).firstMatch
                if cellText.label != "" {
                    name += "-\(cellText.label.replacingOccurrences(of: " ", with: "-"))"
                }
                print("opening \(name)")
                menuItem.element.tap()
                // Screenshot this view and then re-open the menu
                snapshot(name)
                openTheMenu()
            }
        }
    }
}

I'd like to be able to dynamically generate each screenshot as it's own test case so that these will be reported correctly as individual tests, maybe something like:

[T] Screenshots
    [t] testFavouritesViewScreenShot()        ✓
    [t] testGiveFeedbackViewScreenShot()      ✓
    [t] testSettingsViewScreenShot()          ✓

I've had a look at the documentation on creating tests programmatically but I'm not sure how to set this up in a swifty fashion. - Ideally I would use closures to wrap the existing screenshot tests in to their own XCTestCase - I imagined this like the following but there doesn't appear to be any helpful init methods to make this happen:

for menuItem in cells {
    let test = XCTestCase(closure: {
        menuItem.tap()
        snapshot("menuItemName")
    })
    test.run()
}

I don't understand the combination of invocations and selectors that the documentation suggests using and I can't find any good examples, please point me in the right direction and or share any examples you have of this working.

Mccaskill

Since you actually know the difference at compile time, why not just implement a method with common parts a leverage it in your per-case tests? – Downhill

I'm trying to figure out how to write the per case tests dynamically - because I don't know what menu items there will be at compile time. – Mccaskill

Answer 1

You probably can't do it in pure swift since NSInvocation is not part of swift api anymore.

XCTest rely on + (NSArray<NSInvocation *> *)testInvocations function to get list of test methods inside one XCTestCase class. Default implementation as you can assume just find all methods that starts with test prefix and return them wrapped in NSInvocation. (You could read more about NSInvocation here)
So if we want to have tests declared in runtime, this is point of interest for us.
Unfortunately NSInvocation is not part of swift api anymore and we cannot override this method.

If you OK to use little bit of ObjC then we can create super class that hide NSInvocation details inside and provide swift-friendly api for subclasses.

/// Parent.h

/// SEL is just pointer on C struct so we cannot put it inside of NSArray.  
/// Instead we use this class as wrapper.
@interface _QuickSelectorWrapper : NSObject
- (instancetype)initWithSelector:(SEL)selector;
@end

@interface ParametrizedTestCase : XCTestCase
/// List of test methods to call. By default return nothing
+ (NSArray<_QuickSelectorWrapper *> *)_qck_testMethodSelectors;
@end

/// Parent.m

#include "Parent.h"
@interface _QuickSelectorWrapper ()
@property(nonatomic, assign) SEL selector;
@end

@implementation _QuickSelectorWrapper
- (instancetype)initWithSelector:(SEL)selector {
    self = [super init];
    _selector = selector;
    return self;
}
@end

@implementation ParametrizedTestCase
+ (NSArray<NSInvocation *> *)testInvocations {
    // here we take list of test selectors from subclass
    NSArray<_QuickSelectorWrapper *> *wrappers = [self _qck_testMethodSelectors];
    NSMutableArray<NSInvocation *> *invocations = [NSMutableArray arrayWithCapacity:wrappers.count];

    // And wrap them in NSInvocation as XCTest api require
    for (_QuickSelectorWrapper *wrapper in wrappers) {
        SEL selector = wrapper.selector;
        NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];
        NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
        invocation.selector = selector;

        [invocations addObject:invocation];
    }

    /// If you want to mix parametrized test with normal `test_something` then you need to call super and append his invocations as well.
    /// Otherwise `test`-prefixed methods will be ignored
    return invocations;
}

+ (NSArray<_QuickSelectorWrapper *> *)_qck_testMethodSelectors {
    return @[];
}
@end

So now our swift test classes need to just inherit from this class and override _qck_testMethodSelectors:

/// RuntimeTests.swift

class RuntimeTests: ParametrizedTestCase {

    /// This is our parametrized method. For this example it just print out parameter value
    func p(_ s: String) {
        print("Magic: \(s)")
    }

    override class func _qck_testMethodSelectors() -> [_QuickSelectorWrapper] {
        /// For this example we create 3 runtime tests "test_a", "test_b" and "test_c" with corresponding parameter
        return ["a", "b", "c"].map { parameter in
            /// first we wrap our test method in block that takes TestCase instance
            let block: @convention(block) (RuntimeTests) -> Void = { $0.p(parameter) }
            /// with help of ObjC runtime we add new test method to class
            let implementation = imp_implementationWithBlock(block)
            let selectorName = "test_\(parameter)"
            let selector = NSSelectorFromString(selectorName)
            class_addMethod(self, selector, implementation, "v@:")
            /// and return wrapped selector on new created method
            return _QuickSelectorWrapper(selector: selector)
        }
    }
}

Expected output:

Test Suite 'RuntimeTests' started at 2019-03-17 06:09:24.150
Test Case '-[ProtocolUnitTests.RuntimeTests test_a]' started.
Magic: a
Test Case '-[ProtocolUnitTests.RuntimeTests test_a]' passed (0.006 seconds).
Test Case '-[ProtocolUnitTests.RuntimeTests test_b]' started.
Magic: b
Test Case '-[ProtocolUnitTests.RuntimeTests test_b]' passed (0.001 seconds).
Test Case '-[ProtocolUnitTests.RuntimeTests test_c]' started.
Magic: c
Test Case '-[ProtocolUnitTests.RuntimeTests test_c]' passed (0.001 seconds).
Test Suite 'RuntimeTests' passed at 2019-03-17 06:09:24.159.

Kudos to Quick team for super class implementation.

Edit: I created repo with example github

Selectivity

Hey thanks for taking the time, looks like this could work tho as you say, it's not the most Swifty solution, would you mind adding some comments to explain what it's doing at each step? - i'll give this a test later hopefully. – Mccaskill

@Mccaskill I add more information in general and inline comments. Feel free to ask if you still have questions. – Selectivity

[self instanceMethodSignatureForSelector:selector]; is erroring with -> No known class method for selector 'instanceMethodSignatureForSelector:' - also I'm not sure this is going to work without a bridging header? – Mccaskill

@Mccaskill I added link on example project. Try it out. – Selectivity

@Mccaskill And yes, you need bridging header otherwise you can't subclass our ObjC wrapper – Selectivity

OOooo, this even makes the tests show up in the Xcode sidebar! – Analphabetic

One of those rare cases where I miss not having a deep Objective-C knowledge. I'm so glad you wrote this answer, thank you! – Serin