Azure DevOps code coverage with .coverage for PR and Cobertura for full report

Question

I have an Azure DevOps pipeline that validates pull requests. I have configured dotnet test to collect code coverage metrics using the --collect "Code coverage" argument:

- task: DotNetCoreCLI@2
  displayName: dotnet test
  inputs:
    command: 'test'
    arguments: '--configuration $(BuildConfiguration) --collect "Code coverage" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura'
    workingDirectory: $(baseWorkingDirectory)
    projects: 'tests/**/*.csproj'
    nobuild: true

As you can see, I'm also passing /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura. This I have to do in order to generete a coverage report:

- task: CmdLine@2
  inputs: 
    script: dotnet tool install -g dotnet-reportgenerator-globaltool

- task: CmdLine@2
  inputs: 
    script: reportgenerator -reports:$(Build.SourcesDirectory)/tests/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'

Documentation states that code coverage for pull requests is available only using the Visual Studio code coverage results format (file extension .coverage)

So:

• I have to use Visual Studio code coverage --collect:"Code Coverage" to get code coverage for pull requests, because Cobertura format won't work.
• I need to use Cobertura format in order to get a readable report on the Code Coverage tab in the pipeline, but the Cobertura report won't show up if I use --collect:"Code Coverage" at the same time.

It seems I can't get both code coverage for PRs and a full report in Cobertura format at the same time.

Other people seem to have the same problem, but the issue wasn't resolved in that thread.

Am I missing something?

Holofernes

Same problem here, three years later. We have the full coverage (on builds) with coverlet and cobertura with ReportGenerator. But not the diff coverage on PRs. This is frustrating. – Surprisal

Answer 1

Can you try use this?

You have to install coverlet.collector package in all your test projects. Please notice that I used argument --collect:"XPlat Code Coverage".

# You just added coverlet.collector to use 'XPlat Code Coverage'
- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true'
    workingDirectory: $(Build.SourcesDirectory)

- task: DotNetCoreCLI@2
  inputs:
    command: custom
    custom: tool
    arguments: install --tool-path . dotnet-reportgenerator-globaltool
  displayName: Install ReportGenerator tool

- script: ./reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"
  displayName: Create reports

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: $(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml  

Please be aware that you may have different folder strcture.

If you want to use Code coverage for pull requests feature you should not use Cobertura:

Code coverage for pull requests capability is currently only available for Visual Studio code coverage (.coverage) formats. This can be used if you publish code coverage using the Visual Studio Test task, the test verb of dotnet core task and the TRX option of the publish test results task. Support for other coverage tools and result formats will be added in future milestones.

Eigenvalue

But this is exactly the same yml that I have. I also have installed the coverlet.collector package. – Knott

It's not the same. You use --collect "Code coverage". In mine code you have --collect:"XPlat Code Coverage". And this exactly what is written in documentation. So if you have already coverlet.collector can you try with --collect:"XPlat Code Coverage" – Eigenvalue

I see, good catch. However I changed it, and the Code Coverage tab is shown, but not the pull request comment. Do you get both? I mean both the Code coverage tab, and the pull request comment. – Knott

I'm in the same situation. In my project is generated a coverage.cobertura.xml file, so test results are on the build pipeline. Okay but without .coverage file we can't use the "diff coverage" on Pull requests, that will be a great improvement. We want to add a Pull request Coverage Checks policy, but we seems obligates to do a choice. – Surprisal

Answer 2

Unfortunately Azure DevOps is not supporting both code coverage PR support and readable reports at the same time.

My recommendation is to use dynamic code coverage (--collect "Code Coverage") and get *.coverage files. On PR build you can publish test results with publishRunAttachments: true. By this you should get code coverage PR support. Additionally you can use dotnet-coverage to convert your *.coverage reports to cobertura. Then you can use report-generator to generate readable coverage report. This coverage report can be uploaded as zip into artifacts of build. In this case using PublishCodeCoverageResults@1 will not work as Azure DevOps is not supporting it.

On non-PR builds you can again use dynamic code coverage but when publishing test results specify publishRunAttachments: false. Then again use dotnet-coverage to merge and convert your reports to cobertura. Finally you can add PublishCodeCoverageResults@1 task to get readable coverage report.

Both dotnet-coverage and report-generator are cross-plat dotnet global tools. You can easily install it in your pipeline.

Example script to generate html report and zip it:

dotnet-coverage merge -r -f cobertura -o merged.cobertura.xml *.coverage
reportgenerator -reports:merged.cobertura.xml -targetDir coverageReportHtml -reporttypes:HtmlInline
zip -r report.zip coverageReportHtml/*

There is one more workaround but I didn't try this. You could theoretically trigger 1 more build from your PR build. The second build could download *.coverage artifacts from first one and then use tooling like above to finally perform PublishCodeCoverageResults@1 task.

Tacet

Answer 3

In the pipeline definition:

- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    projects: 'path/to/test.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
    arguments: '--collect "Code Coverage" '
    publishTestResults: true

In test project (or Packages.props if using https://github.com/microsoft/MSBuildSdks/tree/main/src/CentralPackageVersions)

...
  <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
...

According to https://github.com/microsoft/vstest/issues/981 and with Microsoft.NET.Test.Sdk 17.0.0+ coverlet is no longer needed as all coverage processing can be done using .coverage files on other-than Windows platforms using --collect:"Code Coverage". coverlet is no longer needed.

Presidentelect

Hi @Stan, But that what it means? We can now use just .coverage file and so validate Pull Requests (as asked by @Kristoffer) and use also the full report on the build pipeline? – Surprisal

Yes, exactly, .coverage can be used on other-than-Windows platforms, and 17.1.0 adds support for Mac OS. – Presidentelect

Okay @Presidentelect , however I think there is something to give up. Because until now I used coverlet and cobertura with ReportGenerator and I have a tab with an HTML report on the Build descriving the entire codebase code coverage. But ReportGenerator is not compatible with .coverage and is not clear for me how I can use on both (diff coverage on PR and full coverage on the build) usign .coverage with ReportGenerator. – Surprisal

You are correct. Things got simplified and all that is needed to get a cross-platform code coverage are: --collect "Code Coverage", Microsoft.NET.Test.Sdk 17.0.0+, and a DotNetCreCLI@2 task with test command and publishTestResults: true. – Presidentelect

Hi @Presidentelect , I tried many times, and now I have Code coverage on Pull Requests with less verbose yaml. I also have the same code coverage accessible from the build (there is a tab called "Code coverage"), but with that awful "Download code coverage results" instead of a visual report I had before with cobertura and ReportGenerator. But ReportGenerator is not compatible with .coverage file. It would be nice to have both and with visual report on the build (is also the request of this Question), but I think is impossible at the moment. – Surprisal

Hi @Presidentelect , how you can see I find a way here, using two pipelines: github.com/microsoft/vstest/issues/981#issuecomment-1034093540 – Surprisal