Xcode UI Testing: Automatically taking snapshot when predicates fail?

Question

Xcode UI Testing takes automatic screenshots for viewing in the results navigator whenever a test fails, which is greatly helpful. However, that does not include tests that fail because a predicate is failed. Since predicates are often for basic checks (such as if an element exists or not on a current view), that is a huge drawback because a screenshot would be useful in diagnosing what was happening in the app when the test failed.

Does anyone know how to force a screenshot? Does this require integrating the Fastlane Snapshot tool?

Greasepaint

Answer 1

On tearDown you can check if test failed (that's helpful if you are not discarding screenshots when tests pass.)

if let failureCount = testRun?.failureCount, failureCount > 0 {
  takeScreenshot()
}

If you are using Xcode 9 already, the takeScreenshot function can use the new API (If not, then use the workaround mentioned by the other answer) :

let screenshot = XCUIScreen.main.screenshot()
let attach = XCTAttachment(screenshot: screenshot)
add(attach)

You can also name the attach and change its lifetime ;)

See Apple's documentation for how to use and where to find them (the "Report navigator" in View > Navigators > Reports) in more detail.

Woo

Do you have any idea where it saves the snapshot? Like the Fastlane snapshot can I set a path for that? Thanks. – Listerism

The attachment can be found under the "Report navigator" (View > Navigators > Reports). Find your scheme's latest build, look under the Tests for failed ones, dig into the tree to find a "Collecting extra data to assist test failure triage" > "Automatic Screenshot" attachment. But your own attachments can also be found in the report, e.g. if I put that code in the Tear Down phase, I can see I "Added attachment named...". – Brinker

Answer 2

You don't have to integrate Fastlane Snapshot for this. The only trick Snapshot is doing to force screenshot is triggering this code:

XCUIDevice.shared().orientation = .unknown

This will not alter UI as described on Snapshot documentation.

Unfortunately this will not work if you're using expectation for your predicate and you put this code into waitForExpectations(timeout:handler:) handler closure and I don't know why.

To workaround this you can create your own XCTestObservation handler like this:

class MockObserver: NSObject, XCTestObservation {
    func testCase(_ testCase: XCTestCase, didFailWithDescription description: String, inFile filePath: String?, atLine lineNumber: UInt) {
        XCUIDevice.shared().orientation = .unknown
    }
}

XCTestObservationCenter.shared().addTestObserver(MockObserver())

You can put this code in either setUp() method or specific test... method.

The test output is a little weird as it will show "Set device orientation to Unknown" as an error and actual predicate error inside but you will have your screenshot:

Schmo

Answer 3

You can override the recordFailure method to capture screenshots on any kinds of failures.

override func recordFailure(withDescription description: String, inFile filePath: String, atLine lineNumber: Int, expected: Bool) {
        add(XCTAttachment(screenshot: XCUIScreen.main.screenshot()))
        super.recordFailure(withDescription: description, inFile: filePath, atLine: lineNumber, expected: expected)
}

Windham