How to include NUnit test output when publishing test results in Azure DevOps?

Question

I have a vNext build which runs a few NUnit tests and publishes the results. However, the output produced by the tests is not published, even though it is found in the respective XML file.

For example:

But, if I inspect the respective XML files - it is there:

Notice the <output> element.

I would very much like to see this output published along the test results. How can I do it?

P.S.

There is another problem with the publishing. The Run Duration is reported as 4h 59m in the top pane, but in the bottom details pane it is the correct 10:48.176 minutes. The 4h 59m looks very much as 5h, which is the time difference between EST and UTC. The tests were run on an Octopus server and were fetched by the vNext build. Maybe there is a time zone confusion somewhere.

EDIT 1

We have an on-premises TFS

EDIT 2

Examining the source code of the Publish Test Results task reveals that it uses Microsoft.TeamFoundation.TestClient.PublishTestResults assembly to parse the NUnit XML test result. Specifically the following C# code is used to parse the test-case element (NUnit3ResultsXmlReader.cs):

if (testCaseResultNode.Attributes["result"] != null)
{
  testCaseResultData.TestCaseResult.Outcome = !string.Equals(testCaseResultNode.Attributes["result"].Value, "Passed", StringComparison.OrdinalIgnoreCase) ? (!string.Equals(testCaseResultNode.Attributes["result"].Value, "Failed", StringComparison.OrdinalIgnoreCase) ? (!string.Equals(testCaseResultNode.Attributes["result"].Value, "Skipped", StringComparison.OrdinalIgnoreCase) ? TestOutcome.Inconclusive.ToString() : TestOutcome.NotExecuted.ToString()) : TestOutcome.Failed.ToString()) : TestOutcome.Passed.ToString();
  XmlNode xmlNode1 = testCaseResultNode.SelectSingleNode("failure");
  if (xmlNode1 != null)
  {
    XmlNode xmlNode2 = xmlNode1.SelectSingleNode("message");
    XmlNode xmlNode3 = xmlNode1.SelectSingleNode("stack-trace");
    testCaseResultData.TestCaseResult.ErrorMessage = xmlNode2 != null ? xmlNode2.InnerText : (string) null;
    testCaseResultData.TestCaseResult.StackTrace = xmlNode3 != null ? xmlNode3.InnerText : (string) null;
    XmlNode xmlNode4 = testCaseResultNode.SelectSingleNode("output");
    if (!string.IsNullOrWhiteSpace(xmlNode4 != null ? xmlNode4.InnerText : (string) null))
      testCaseResultData.ConsoleLog = xmlNode4.InnerText;
  }
}

From which it follows that the assembly authors think that test output is only useful when the test-case in question has failed. This is an unfortunate decision, because it is not up to them to decide when the output is useful. If it is in the test results XML, then it should be published.

Opened a new issue https://github.com/Microsoft/azure-pipelines-tasks/issues/8979

Opened an Azure DevOps feature request - https://developercommunity.visualstudio.com/idea/432166/the-publish-tests-azure-devops-plugin-should-publi.html

Ericson

What do you mean by 'output'? I can see the tests listed and the results from it. – Hymenopterous

The test in question produces console output. This output is produced even if the test passes. And this output is included in the XML. But, it is nowhere to be found in the test page in the build. – Ericson

Answer 1

My solution to this unfortunate behavior force upon us is to mangle the XML test result before handing it off to the Publish Test Results task.

The following powershell code does the trick:

$Modified = $false
$xml = [xml](cat $OriginalTestResultsXmlFile -Raw)
$xml.SelectNodes('//test-case') |? { !$_.failure -and $_.output } |% { 
    $Modified = $true
    $_.InnerXml = @"
<!-- Workaround the issue https://github.com/Microsoft/azure-pipelines-tasks/issues/8979. No real failure here -->
<failure/>
$($_.InnerXml)

"@
}

if ($Modified)
{
    $xml.Save($TestResultsXmlFile)
}
else
{
    move $OriginalTestResultsXmlFile $TestResultsXmlFile
    $OriginalTestResultsXmlFile = $TestResultsXmlFile
}

We have to trick the task into executing the logic we need by creating a dummy failure element besides the output element.

Lo and behold:

Ericson