Does anyone know what happened to test results for Xcode 11 UI testing?

Question

I used to be able to access results in Derived Data inside the .xcresults package. .xcresults package still exists, but where there used to be a file named "TestSummaries.plist", and subdirectories containing attachments (like screenshot files, etc), there is now a subdir named "Data" which contains binary format files. So now I can no longer programmatically extract screenshot files, or upload test results to a database like qTest or ALM.

I have confirmed that Xcode 10 still outputs results in the old format. I have searched on my machine for the "TestSummaries.plist" file or similar artifacts thinking that perhaps they would be in a different location. No luck so far.

Impignorate

Answer 1

Apple changed the format of xcresult starting in Xcode 11. In order to get items out of it now, you have to go through Xcode's xcresulttool (xcrun xcresulttool), but it can be entirely cumbersome as you have to go through multiple layers of data structures in the new tree structure just to get to your attachments (which can be hundreds of commands in xcresulttool if you have a large amount of tests).

My workplace open sourced a tool called xcparse that allows you to export out the screenshots from the new format (https://github.com/ChargePoint/xcparse). You can install it Homebrew: brew install chargepoint/xcparse/xcparse It simplifies the process by automating the xcresulttool calls & parsing you'd have to do.

Eachern

fantastic. thanks for sending this. I played with the xcresulttool, but it seemed that the output did not have what I needed. I will give your xcparse tool a look. – Impignorate

@KerwinLumpkins: If there's something particular you need, feel free to add an issue request. We focused mostly on extracting screenshots/attachments as that was our need for our CI, but are open to adding more functionality. – Eachern

As much as I like this xcparse tool and the work put into it, I'm asking is there still no simpler way to get the screenshots without extra tools in the CI chain? How does make Apple internally their screenshots for their apps? By the way nice explained here by you guys: chargepoint.com/engineering/xcparse – Formulism

@BrunoBieri Apple engineers have access to devices that basically run jailbroken (See tidbits.com/2020/08/04/…). Due to this, they're not really using the xcresult's screenshot feature for their App Store screenshots as they can just grab the screenshot directly from the device & run internal tools to modify the status bar and other properties. Definitely agree Apple should make this easier & part of xcresulttool natively; join me in filing a Feedback Assistant ticket with them! – Eachern

Answer 2

Go to the “Report Navigator”, right click on the test in question, and choose “Show in Finder”:

In Finder, you’ll see an entry like Test-MyApp-2019.10.14_13-12-40--0700.xcresult, one for each test run.

In Finder, you can also right-click and choose “Show Package Contents” to see the individual files. By the way, this “Report Navigator” also gives a nice GUI for perusing the test results without going through the files, too.

Ribose

Thanks, but that doesn't get me what I was looking for. I am aware of the .xcresult file, and how to open it. (by the way, with Xcode 11, you can now double click on the .xcresult file to open without using the Show Package Contents). My goal however is to programmatically read the results file, which used to be a file named "TestSummaries.plist". That file no longer exists in the results, and the Attachments subdirectory is gone now too. If someone knows how to open the binary files (file with a name like data.0~-TzUFRo-...==) I'm wide open to being educated. – Impignorate