Running individual XCTest (UI, Unit) test cases for iOS apps from the command line

Question

Is it possible to run individual test cases, or individual test suites, from an iOS app test target, instead of all the test cases, from a command line interface?

You can run tests from command line with xcodebuild, out of the box. When you do so, you run all of the test cases contained in the test target you've selected.

You can also do so with scan from Fastlane, though I believe you're restricted to running all of the tests of the build scheme you select (as above), so it's not different from xcodebuild.

You can run specific tests with xctool from Facebook, but it doesn't use xcodebuild, and is restricted to running on simulators only, not actual iOS test devices.

I found a reference to running the xctest command line utility directly, but it seems to be an undocumented feature and targets DerivedData. This is complicated by the fact that UI Tests, have their *xctest files in a separate XCTRunner bundle.

Squaw

should you have an example with xctool to run specific UI Tests? – Traject

@Traject xctool has the "-only" option to let you specify individual test files (suites) and test methods (cases) to run at a time. It works for unit tests and I assume it does the same for UI tests. I haven't tried it myself, though, because I require a testing solution to run on a real device, not on simulators only, so xctool is not a solution. – Squaw

xctool doesn't support UI test, see this answer: #35181626 – Traject

Thanks for the heads up. I would have thought that it's possible to trigger Xcode UI Tests the same way we run Unit Tests, but I guess I'm wrong. – Squaw

Answer 1

It is now possible with Xcode 8 using the -only-testing parameter with xcodebuild:

xcodebuild test -workspace <path>
                -scheme <name>
                -destination <specifier>
                -only-testing:TestBundle/TestSuite/TestCase

Traject

Perfect. Thanks for the screenshot. It's also worth mentioning, as it was news to me, that we've also got new xcodebuild actions to split out the components of the original test action - namely build-for-testing and test-without-building. The latter being handy if you want to run a test multiple times without the overhead of building each time. Also great for aligning with Maven lifecycle phases if you need that. – Altdorf

xcodebuild test -workspace <path> -scheme <name> -destination <specifier> -only-testing:TestBundle/TestSuite/TestCase – Mckie

Is it possible to filter using asterisks? – Torrez

Answer 2

You can edit the scheme to run only specific tests. Select the scheme, then edit scheme. In the appearing window, select the Test phase and disable/enable individual tests.

You can also add schemes to run subsets of tests. When running the tests from command line you can specify the scheme to use for the test (at least in fastlane).

Upshaw

for 100 subsets of tests, you will have 100 schemes. No a good simulator. I prefer the other one with environment variable. – Traject

Answer 3

Run an individual XC test

To run an individual test case you can use -only-testing

-only-testing Pattern

-only-testing:<target>/<class_name>/<test_name>

xcodebuild Pattern

xcodebuild test 
-workspace "<name>.xcworkspace"  
-scheme "<name>" 
-destination '<options>' 
-only-testing "<test_case>"

Example

//for example(several test cases)
xcodebuild test 
-workspace "MyApp.xcworkspace"
-scheme "MyAppTest" 
-destination 'platform=iOS Simulator,name=iPhone 11 Pro Max,OS=13.5' 
-only-testing "MyAppTest/TestClass/foo"
-only-testing "MyAppTest/TestClass/foo2"

For example if Test Navigator looks like

the parameter will have the following type

-only-testing:F49088168M_OBDIITests/HexUtilityTests/testHexToBinStringFormat

If you want to add an additional test case you can add one more -only-testing

Also you can skip a test using: -skip-testing

Test results you can find in Derived Data^[About]

<derived_data>/<project_name>-dzqvyqfphypgrrdauxiyuhxkfxmg/Logs/Test/Test-<target_name>-<date>.xcresult

[Xcode screenshot]

Injurious

Answer 4

I was in similar situation as you and have built a python script that triggers the set of test case/s that i want. Its a little bit elaborate process but works for me and has been very useful over time in implementing DataProvider methods, Rerunning of failed test cases and other customizations I required.

Some relevant steps for what you want to achieve.

1. Override testInvocations method present in XCTestCase to do below steps
   • In this method I read Environment Variable XXXX.
   • This environment variable is basically a comma separated test case method names.
   • Now create NSInvocations for each test method that you want to trigger.
   • Return array of Invocations.
2. How to pass Environment Variable?
   • In scheme add an environment variable named XXXX.
   • Scheme files are standard xml files, write a script that modifies the scheme file to contain the Comma separated values in environment variable.

If you require more info add a comment I will reply to it.

Carroty

Answer 5

To run an individual test or test class you can click the diamond next to it in the gutter. This is right next to where line numbers appear if you have them turned on.

In this screenshot my mouse is hovering over the diamond. Notice how it has changed to a little play arrow indicating it will be run.

You can then re-execute the most recently run test(s) with ⌃⌥⌘ G.

As far as I know this cannot be done via the xcodebuild.

Karyolymph

I've clarified the original question. My main intent was asking if individual test cases can be triggered from command line, so there could be continuous integration that only calls a subset of tests instead of the entire target. – Squaw

See @dasdom's answer. There is some discussion going on in the Swift mailing list that is aiming to add this exact feature. – Karyolymph

While you didn't answer the OP's question, this was exactly my question, I really appreciate your answer! – Erbe

Answer 6

May be this helps. I was ignorant because I didn't know that you can disable some particular tests in Test navigator by right-clicking them.

Then you run xcodebuild test -scheme <name> as usual, and all the disabled tests will be ignored. Worked with Fastlane for me.

Dollydolman

Answer 7

If you don't have .xcworkspace. Use -project instead.

xcodebuild test \
-project "name.xcodeproj" \
-scheme "scheme name" \
-destination 'platform=iOS Simulator,name=iPhone 14 Pro,OS=16.4' \
-only-testing:<test target name>/<test class name>/<test function name>

Cambrian