Running a single KIWI spec with xctool

Question

Has anyone been able to successfully pass the KW_SPEC variable to xctool. I'm trying to run a single KIWI spec by using https://github.com/kiwi-bdd/Kiwi/wiki/Kiwi-FAQ#q-how-do-i-run-a-single-spec-describecontextit.

I can run all the tests successfully with xctool but it doesn't seem to pick up the KW_SPEC value. I've tried it in a lot of different places with the command line but no luck.

e.g.:

xctool -destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
    -sdk iphonesimulator -workspace SampleProject.xcworkspace 
    -scheme SampleProject KW_SPEC=NewAssessmentTests.m:12 test 
    -only SampleProject_Acceptance_Tests

Using Kiwi v2.3.1 and xctool 0.2.3

Cheers, Mo

Parlormaid

Have you ever succeeded in this? I'm trying to do the same with no luck. From reading the Kiwi code I gather: (1) KW_SPEC is retrieved using [[[NSProcessInfo processInfo]environment] which means build-time settings like you used above won't work - I believe you need to set it as an environment variable in the scheme (see nshipster.com/launch-arguments-and-environment-variables) (2) The filename and line are compared against an atos result which appears consistent with the format you used, but when I tried it myself nothing ran (0 tests executed) – Beauty

Related source files: https://github.com/kiwi-bdd/Kiwi/blob/ae9f6d83faf774754ea60a9dfc6556eb23800f47/Classes/Core/KWExampleSuiteBuilder.m, https://github.com/kiwi-bdd/Kiwi/blob/ae9f6d83faf774754ea60a9dfc6556eb23800f47/Classes/Core/KWSymbolicator.m. Related GitHub issue: github.com/kiwi-bdd/Kiwi/issues/614. – Beauty

Answer 1

Update 11/03/2015

@OhadSchneider's comment made me realise that KW_SPEC worked for me because I had set it in the test configuration for my scheme (Edit scheme->Test->Arguments->Environment variables). Setting the variable from shell doesn't work as that variable applies only to the actual build, and not when executing the unit test target.
But there's a workaround to this, by modifying the test phase of your scheme and adding a KW_SPEC environment variable with the value $KW_SPEC, this will expand when running xcodebuild to the value passed to the xcodebuild command (as in my original answer). After this, xcode will will gracefully use the passed KW_SPEC variable, xctool still has the skipped teste marked as failure issue.

Original answer

If you want KW_SPEC as an environment variable to xctool (or to any *nix tool), then you have to place it before the command, otherwise it will be considered a build setting:

KW_SPEC=NewAssessmentTests.m:12 xctool 
    -destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
    -sdk iphonesimulator -workspace SampleProject.xcworkspace 
    -scheme SampleProject test 
    -only SampleProject_Acceptance_Tests

This will however lead to another problem: xctool will report as errored the tests that don't run, and will report the test as failed, even if no tests have failed. xcodebuild doesn't have this problem as it either doesn't do unit tests discovery, or ignore tests that didn't run, a thing that xctool fails to do.

Syncope

It doesn't work even with xcodebuild (actually even when I define it inside XCode, the spec simply doesn't run), but the bounty is about to expire so I might as well award it for effort :) – Beauty

@OhadSchneider thanks for checking it, I rerun on my machine and it still works (cliche doesn't it :) The problem was the fact that I altered my scheme from Xcode to pass that environment variable, making the command line one useless. I will update my answer with this, as I found the workaround for xcodebuild. – Syncope

Thanks for following up, but I was actually aware of that. My problem is that the spec simply doesn't run (I know it reads the flag becuase when it's defined nothing runs, when not everything runs). I'm using the line number where SPEC_BEGIN is defined. Anyway relying on line numbers is probably too fragile anyway, so I'll just use a different target for each spec. Too bad using the name of the spec doesn't work. – Beauty

Hmm.. I used the line where the it I wanted to run began, and I got running only that test only, I don't think using the line where with SPEC_BEGIN would help... – Syncope

In the FAQ, they claim spec/describe/context/it are all supported: github.com/kiwi-bdd/Kiwi/wiki/Kiwi-FAQ. I think it failed for me too, but if you get the chance it would be interesting to know whether the SPEC_BEGIN line works for you (may be something in our build that makes atos fail). – Beauty

Yeah, it looks like it works for describe/context/it, but it doesn't work for SPEC_BEGIN. Wrapping all tests of a spec in a describe should work. – Syncope

Thanks for confirming. Like I said I've deemed this as too fragile for my taste, but it's good to know. – Beauty

Hi Ohad and Cristik, thanks for following up with these answers. I still didn't have any luck with the command line and xctool – Parlormaid

@MoJaimangal are you still not able to run the tests that you want, or did you run into the other problem of xctool - reporting as failed test that didn't run? – Syncope

Works for Xcode if one sets the environment variable to filename.m:line as described in the Kiwi FAQ. Too bad one has to manually edit the scheme to achieve that. – Talley