How to fetch and parse all the generated coverage.cobertura files in CI pipelines?

Question

Given a .Net 5 solution with multiple xUnit test projects I can run dotnet test from the root of the solution and it will run all the tests.

I would like to generate reports so based on https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=windows#integrate-with-net-test I run dotnet test --collect:"XPlat Code Coverage" which generates a coverage.cobertura file per test project.

Based on https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=windows#generate-reports I know that I can install the dotnet-reportgenerator-globaltool tool and get a visual HTML report based on each coverage.cobertura file.

But I want to add a CI pipeline where I want to make the pipeline fail when the line coverage is below x %.

Given the following sample Gitlab CI configuration

image: mcr.microsoft.com/dotnet/sdk:5.0

stages:
  - tests

tests:
  stage: tests
  script:
    - dotnet test --collect:"XPlat Code Coverage"

how can I gather all the generated coverage.cobertura.xml files, read the line coverage and let the pipeline fail if the value is below e.g. 80%?

Example:

tests:
  stage: tests
  script:
    - dotnet test --collect:"XPlat Code Coverage"
    # for each coverage.cobertura file found in the test projects
    # parse the file
    # read the line coverage
    # fail if the value is less than 80 percent

It would be nice if I don't have to reinvent the wheel if tools like xUnit already provide such functionality!

---

Edit:

I know that I could also use the allow_failure keyword to leave this stage in a warning state. This would be fine for me, I just want to know how to read the required information from the generated reports and validate them to decide if that stage should pass, fail or be unstable.

Iseabal

I mean, a way of doing this, is to calculate the value itself and if it is below threshold x, perform "exit 1" – Goatfish

@Goatfish yes, but how would I calculate it? How can I search for all the line coverages in all generated files? – Iseabal

I would suggest you have a look here: learn.microsoft.com/en-us/dotnet/core/testing/… This may help you in your struggle. Once your file is converted to for example an html file, you might be able to parse it, extract the number you are looking for and perform an exit 1 if needed – Goatfish

Answer 1

This might sound like a really novice approach, but here is something that worked for me. I am using Azure devops and I had similar multiple projects, resulting in multiple coverage.cobertura file per test project

First I used Report Generator task to merge all coverage reports into one, and store it in working directory. Below is the yaml. I'm generating both, HtmlFormat as well as Cobertura report

- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
  displayName: ReportGenerator
  inputs:
    reports: '$(Build.SourcesDirectory)/**/*cobertura.xml'
    targetdir: '$(System.DefaultWorkingDirectory)/CoverageResults'
    reporttypes: 'HtmlInline_AzurePipelines;Cobertura'

Resulting cobertura report looks like this:

After this, I tried reading this merged xml report using powershell task

- powershell: |
   [XML]$coverage = Get-Content $(System.DefaultWorkingDirectory)/CoverageResults/Cobertura.xml
   if($coverage.coverage.'line-rate' -ge .50){Write-Host "The value is greater than 50."}else{throw}
  displayName: 'PowerShell Script'

In this task, I'm trying to read the coverage file, and then accessing the line-rate attribute. If line rate of merged report is not greater than or equal to 50, I'm throwing error. With ErrorActionPreference set to Stop for my powershell task, the pipeline stops if line coverage is not as per expectation. You can also include other conditions like branch rate for better accuracy .

There is a lot to improve in this I guess, but this was just a quick workaround that I could come up with. Please feel free to suggest improvements

Fowlkes

Answer 2

According to Coverlet options supported by VSTest integration, XPlat code coverage tool doesn't support merging of coverage report files and validating the threshold yet, but they are working on it. For now there's a solution to merge the reports and calculate the threshold provided by Daniel Paz

They use merge with in VSTest version of the tool with coverlet.collector:

# set MergeWith value in runsettings.xml
$runsettingsFile = "$pipelineFolder/runsettings.xml"
$coverageJsonFullPath = "$testResultsDirectory/coverage.json"
(Get-Content $runsettingsFile).Replace('#mergewith#', $coverageJsonFullPath) | Set-Content $runsettingsFile

# calculate code coverage
Get-ChildItem -Path "test/unit","test/integration","test/component" -Recurse -Filter *.csproj | 
Foreach-Object {
  $dir = "$testResultsDirectory/$($_.BaseName)"
  dotnet add  $_.FullName package coverlet.collector -v 1.1.0
  dotnet test $_.FullName --collect:"XPlat Code Coverage" --settings $runsettingsFile --no-build --logger trx --results-directory $dir
  Copy-Item -Path "$dir/*/coverage.json" -Destination $coverageJsonFullPath -Force
}

# copy results to $testResultsDirectory
Copy-Item -Path "$dir/*/coverage.cobertura.xml" -Destination $testResultsDirectory
Copy-Item -Path "$dir/*/coverage.opencover.xml" -Destination $testResultsDirectory

runsettings:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>opencover,json,cobertura</Format>
                    <MergeWith>#mergewith#</MergeWith>
                    <Exclude>[*]*Migrations*</Exclude> <!-- [Assembly-Filter]Type-Filter -->
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                    <ExcludeByFile>**/Program.cs,**/test/**/*</ExcludeByFile> <!-- Absolute or relative file paths -->
                    <SingleHit>false</SingleHit>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
    <InProcDataCollectionRunSettings>
        <InProcDataCollectors>
            <InProcDataCollector assemblyQualifiedName="Coverlet.Collector.DataCollection.CoverletInProcDataCollector, coverlet.collector, Version=1.1.0.0, Culture=neutral, PublicKeyToken=null"
                                 friendlyName="XPlat Code Coverage"
                                 enabled="True"
                                 codebase="coverlet.collector.dll" />
        </InProcDataCollectors>
    </InProcDataCollectionRunSettings>
</RunSettings>

And check the merged file for a threshold:

# compare code coverage to $coverageThreshold
$coverage =  Select-Xml -Path "$testResultsDirectory/coverage.cobertura.xml" -XPath "/coverage/@branch-rate" | % {[double]::Parse($_.Node.Value) * 100}
Write-Output "Coverage is $coverage"
if ($coverage -lt $coverageThreshold) {
  throw "Code coverage $coverage is less than threshold $coverageThreshold"
}

Montano