Where do Xcode Bots put their results, so I can parse them?

Question

Our dev team has always used Jenkins for our iOS builds, and used Philips Hue lights to inform the team when the build is Building(Yellow), Successful(Green), Failed(Red).

Now we have moved to Xcode CI and Bots, and I do not know when any unit tests fail. We don't even know if the build phase failed.

On Xcode Bots CI you get this "bigscreen" feature: In Apple's "Manage and Monitor Bots from a Web Browser" Docs, you can see that it has all sorts of states that could key up a hue light.

I really don't want to hack something up and parse an HTML page. Although fun, the work does not last long if Apple updates their HTML markup.

Is there a parseable file that is produced when a Xcode bot finishes its integration?

I'd love to get the Hue showing:
* Blue for Analysis warnings
* Orange for Build Warnings
* Red for Build Errors
* Yellow for build running

Abeokuta

Answer 1

I'd like to share the team's solution. We found the place where the Bot Results are stored, we parse it using bash, and send messages to the HUE light via curl system calls. We call the script in a scheme pre and post Build scripts.

We parse the bot's result plist at:

/Library/Server/Xcode/Data/BotRuns/Latest/output/xcodebuild_result.bundle/Info.plist

There you can find all sorts of cool data to use!:

<dict>
    <key>AnalyzerWarningCount</key>
    <integer>0</integer>
    <key>AnalyzerWarningSummaries</key>
    <array/>
    <key>ErrorCount</key>
    <integer>0</integer>
    <key>ErrorSummaries</key>
    <array/>
    <key>LogIdentifier</key>
    <string>705bffcb-7453-49ba-882f-80e1218b59cf</string>
    <key>LogPath</key>
    <string>1_Test/action.xcactivitylog</string>
    <key>Status</key>
    <string>IDEActionResultStatus_Succeeded</string>
    <key>TestFailureSummaries</key>
    <array/>
    <key>TestSummaryIdentifier</key>
    <string>a1554874-4d40-4e94-ae89-a73184ec97a9</string>
    <key>TestSummaryPath</key>
    <string>1_Test/action_TestSummaries.plist</string>
    <key>TestsCount</key>
    <integer>185</integer>
    <key>TestsFailedCount</key>
    <integer>0</integer>
    <key>WarningCount</key>
    <integer>0</integer>
    <key>WarningSummaries</key>
    <array/>
<dict>

• AnalyzerWarningCount
• ErrorCount
• WarningCount
• TestsFailedCount

Oh bash, my sometimes lover, come save the day again.

Also notice the use of the Plist Buddy for parsing Xcode's XML property list files. The primo choice for getting information in and out of plist files.

    #!/bin/bash
    #
    #   By Phil
    #
    exec > /tmp/my_log_file.txt 2>&1
    TEST_RESULT_PLIST="/Library/Server/Xcode/Data/BotRuns/Latest/output/xcodebuild_result.bundle/Info.plist"

    hue_light_green=false

    echo "testResultParse_OwlHue"

    #If not bot, return
    if [ "$(whoami)" != "_teamsserver" ]; then
        echo "$(whoami) - Not a bot!";
        exit 1
    fi

    #1 If file not found ERROR
    if [ ! -f $TEST_RESULT_PLIST ]; then
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"effect\":\"none\",\"hue\":150,\"sat\":255,\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
        echo "Test Result Plist not Found";
        exit 1
    fi

    #2 AnalyzerWarningCount BLUE
    AnalyzerWarningCount=$(/usr/libexec/PlistBuddy -c "Print :AnalyzerWarningCount" "${TEST_RESULT_PLIST}")
    if [ $AnalyzerWarningCount != 0 ]; then
        echo "AnalyzerWarningCount";
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"xy\":[0.16, 0.1],\"hue\":15815,\"sat\":255,\"effect\":\"none\",\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
    fi

    #3 WarningCount
    WarningCount=$(/usr/libexec/PlistBuddy -c "Print :WarningCount" "${TEST_RESULT_PLIST}")
    if [ $WarningCount != 0 ]; then
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"xy\":[0.58, 0.41],\"hue\":15815,\"sat\":255,\"effect\":\"none\",\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
        echo "WarningCount";
    fi

    #4 ErrorCount || TestsFailedCount ERROR
    ErrorCount=$(/usr/libexec/PlistBuddy -c "Print :ErrorCount" "${TEST_RESULT_PLIST}")
    if [ $ErrorCount != 0 ]; then
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"effect\":\"none\",\"hue\":150,\"sat\":255,\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
        echo "ErrorCount";
        exit 1
    fi

    #5 TestsFailedCount ERROR
    ErrorCount=$(/usr/libexec/PlistBuddy -c "Print :ErrorCount" "${TEST_RESULT_PLIST}")
    if [ $TestsFailedCount != 0 ]; then
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"effect\":\"none\",\"hue\":150,\"sat\":255,\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
        echo "TestsFailedCount";
        exit 1
    fi

    #6 None of the above.  SUCCESS
    if [ "$hue_light_green" = true ] ; then
        echo "SUCCESS";
        curl -X PUT -d "{\"on\":true,\"bri\":32,\"effect\":\"none\",\"hue\":25500,\"sat\":255,\"alert\":\"lselect\"}" ipaddress/api/testestest/lights/3/state
    fi

• AnalyzerWarningCount Blue
• ErrorCount Red
• WarningCount Orange
• TestsFailedCount Red

Now when we get a count for any of the above, we get a blinking color change. For example, the following produces a bright blue from our hue:

Abeokuta

Has the location of the results plist file changed in OS X 10.10 Yosemite? I don't even have a folder at /Library/Server/Xcode. – Misquote

Server moved to /Library/Developer/XcodeServer/, but I don't see anything like BotRuns/Latest in there... – Homeopathic

Answer 2

The result path for OS X Server 4.0 seems to be:

/Library/Developer/XcodeServer/IntegrationAssets/Your_Bot/

• Archive.xcarchive.zip
• build.log
• buildService.log
• Your_Bot.ipa
• sourceControl.log
• xcodebuild_result.bundle.zip

The xcodebuild_result.bundle is a zip file now, I parse the build result from buildService.log instead:

Build results summary: {
analyzerWarningChange = 14;
analyzerWarningCount = 14;
errorChange = 0;
errorCount = 0;
improvedPerfTestCount = 0;
regressedPerfTestCount = 0;
testFailureChange = 0;
testFailureCount = 0;
testsChange = 0;
testsCount = 0;
warningChange = 20;
warningCount = 20;
}

Valadez

Worked for me, thanx. Do you have any idea where to find detailed output for each test ? – Aiello

Answer 3

To anyone looking here after these answers, the scraping of the buildService.log files is not necessary (and actually won't even work because of a chicken/egg problem of when the logs get created relative to running the triggers). Try running the command env in the Trigger Script and you'll see that Xcode actually sets environment variables with the results of the test, with some of the more notable ones being:

XCS_BOT_NAME=My New Bot
XCS_WARNING_CHANGE=0
XCS_INTEGRATION_RESULT=succeeded
XCS_TEST_FAILURE_COUNT=0
XCS_TEST_FAILURE_CHANGE=0
XCS_ERROR_COUNT=0
XCS_ANALYZER_WARNING_COUNT=0
XCS_TESTS_CHANGE=0
XPC_SERVICE_NAME=0
XCS_ERROR_CHANGE=0
XCS_WARNING_COUNT=0
XCS_TESTS_COUNT=3
XCS_INTEGRATION_NUMBER=1

Stupe