coverlet does not create report in azure devops

Question

I am using

• coverlet.msbuild v3.2.0
• coverlet.collector v3.2.0

and dotnet sdk v6.0.402

When I run this test command in powershell (restore and build ran before that)

dotnet test --no-build --no-restore --collect:"XPlat Code Coverage" /p:Configuration=$Cfg /p:CollectCoverage=true /p:CoverletOutput=.\CodeCoverage\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"

The report files

• coverage.cobertura.xml
• coverage.opencover.xml

are created and the powershell output is this:

Test run for C:\Users\MyUser\repos\My.Project\Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Attachments:
  C:\Users\MyUser\repos\My.Project\Specs\TestResults\dbe8cb53-3ec5-4227-a231-3bfedf94694f\coverage.cobertura.xml
Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14, Duration: 910 ms - My.Project.Specs.dll (net472)

Calculating coverage result...
  Generating report '.\CodeCoverage\coverage.cobertura.xml'
  Generating report '.\CodeCoverage\coverage.opencover.xml'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| My.Project           | 20.23% | 18.53% | 20.09% |
+----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+
| Average | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+

I have this azure-pipeline task:

- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    arguments: '--no-restore --no-build --collect:"XPlat Code Coverage" /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true  /p:CoverletOutput=$(Build.SourcesDirectory)\CodeCoverage --% /p:CoverletOutputFormat=\"cobertura,opencover\"'
  publishTestResults: true

Which executes this command:

C:\agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\agent\_work\_temp --no-restore --no-build "--collect:XPlat Code Coverage" /p:Configuration=Release /p:CollectCoverage=true /p:CoverletOutput=C:\agent\_work\9\s\CodeCoverage --% "/p:CoverletOutputFormat=\cobertura,opencover\""

And has this output

Test run for C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\a4203b08-db42-449b-86d7-55cb48c54fc4\a4203b08-db42-449b-86d7-55cb48c54fc4\assembly\dl3\e501d05b\fee426f7_ddf4d801\My.Project.Specs.dll
-> Using specflow.json
-> LivingDocPlugin: Output generated in: C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TestExecution.json
Results File: C:\agent\_work\_temp\BUILDMACHINE01$_BUILDMACHINE01_2022-11-10_09_25_37.trx

Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14, Duration: 985 ms - My.Project.Specs.dll (net472)

Attachments:
  C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
Async Command Start: Publish test results
Publishing test results to test run '1436028'.
TestResults To Publish 12, Test run id:1436028
Test results publishing 12, remaining: 0. Test run id: 1436028
Published Test Run : https://dev.azure.com/orgteamservices/My.Project/_TestManagement/Runs?runId=1436028&_a=runCharts
Async Command End: Publish test results
Finishing: Test

No reports are generated on the on-prem agent machine in the expected directory. The agent does create this though C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml

Why is it not creating both reports in the expected directory?

Hume

We have the same problem. Though we didn't have the --collect "XPlat Code Coverage" argument yet. But before we updated the build tools to the latest version 17.4 this worked fine as well. But since the update of the build tools on our build server it no longer generates a coverage file. So I suppose they also updated the build tools for Azure as well since the release of the new .net 7 release. – Unijugate

I've just reverted to the previous build tools version 17.3.6. And it is now working again. So definitely something to do with the upgrade. – Unijugate

This is the reason why our GIthub Action danielpalme/[email protected] failed. Just in case someone is googling "The report file pattern '**/coverage.opencover.xml' found no matching files." – Formality

Answer 1

The problem:

Since Visual Studio Build tools 17.4.0 any arguments (/p:) passed to coverlet are ignored.

The workaround:

Specifying publishTestResults: true as input parameter for DotNetCoreCLI@2 task (or no publishTestResults input at all) causes the following arguments to be added: --logger trx --results-directory $(Agent.TempDirectory). This is normal behaviour btw.

This in turn causes the coverage results to be placed in the agents temp directory as the coverlet arguments are ignored.

So set publishTestResults to false

Add the arguments which are now omitted yourself: --logger trx --results-directory CodeCoverage

Pass any additional runsettings either by file or by adding the following runsettings arguments: --DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover,cobertura

This should mitigate the problem for now.

This is what my entire yml task looks like:

- task: DotNetCoreCLI@2
      displayName: 'Run acceptance tests'
      inputs:
        command: 'test'
        projects: './src/Service.Requirements/Service.Requirements.csproj'
        workingDirectory: './src/Service.Requirements'
        publishTestResults: false
        arguments: '--logger trx --results-directory CodeCoverage --configuration $(buildConfiguration) --no-build --collect "XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude=[Service.Data]*'

Unijugate

"any arguments (/p:) passed to coverlet are ignored" why would MS do this? (apart from to break most of my Azure DevOps pipelines /s) – Quincey

You just need to install nuget Coverlet.MsBuild so /p won't be ignored ;) – Partain

Answer 2

Ok, found a solution.

I have created a coverlet.runsettings file

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>./CodeCoverage/</ResultsDirectory>
  </RunConfiguration>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura,opencover</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

and changed the task like this

diff --git a/azure-pipelines.yml b/azure-pipelines.yml
-            sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/CodeCoverage/coverage.opencover.xml
+            sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/**/coverage.opencover.xml

-          arguments: --no-restore --no-build --collect:"XPlat Code Coverage" /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true  /p:CoverletOutput=$(Build.SourcesDirectory)\CodeCoverage\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"
+          arguments: --no-restore --no-build --collect:"XPlat Code Coverage" --settings ./My.Project.Specs/coverlet.runsettings /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true  /p:CoverletOutput=$(Agent.TempDirectory)\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"

-          summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/coverage.cobertura.xml'
+          summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'

I specify the paths to the report files using the agents temp directory and a glob pattern.

This task creates this output now (abbreviated):

C:\agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\agent\_work\_temp --no-restore --no-build "--collect:XPlat Code Coverage" --settings ./My.Project.Specs/coverlet.runsettings /p:Configuration=Release /p:CollectCoverage=true /p:CoverletOutput=C:\agent\_work\_temp\ --% "/p:CoverletOutputFormat=\cobertura,opencover\""
Test run for C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\fa03db93-d250-4d1a-acfc-4ec086feac88\fa03db93-d250-4d1a-acfc-4ec086feac88\assembly\dl3\6a2f2724\d56f5937_94f5d801\My.Project.Specs.dll
-> Using specflow.json
-> LivingDocPlugin: Output generated in: C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TestExecution.json
Results File: C:\agent\_work\_temp\BUILDMACHINE01$_BUILDMACHINE01_2022-11-11_07_10_08.trx

Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14, Duration: 876 ms - My.Project.Specs.dll (net472)

Attachments:
  C:\agent\_work\_temp\c68cb66e-3fd4-498d-b3e3-218ed75f68a8\coverage.cobertura.xml
  C:\agent\_work\_temp\c68cb66e-3fd4-498d-b3e3-218ed75f68a8\coverage.opencover.xml

It still ignores all the directories I provide but creates both reports in the temp directory where I can access them.

Hume

Another option is to change PublishTestResults to false. This will cause the CLI no longer to add --logger trx --results-directory $(Agent.TempDirectory). Now you can specify your own results directory. When you use sonar cloud don't forget to add extra properties to the SonarCloudPrepare@1 task: sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml – Unijugate

Answer 3

Please include below steps:

It worked for me.

steps:
  - task: UseDotNet@2
    displayName: Install .NET Core 3.1 SDK
    inputs:
      version: '3.1.x'
      packageType: sdk

  - task: DotNetCoreCLI@2
    displayName: 'Run Unit Tests'
    condition: succeeded()
    inputs:
      projects: 'tests/**/*.csproj'
      arguments: '--logger trx --configuration Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:ExcludeByFile="**/*.cshtml" --collect "Code Coverage"'
      command: test
      publishTestResults: true

Abulia

Remember that Stack Overflow isn't just intended to solve the immediate problem, but also to help future readers find solutions to similar problems, which requires understanding the underlying code. This is especially important for members of our community who are beginners, and not familiar with the syntax. Given that, can you edit your answer to include an explanation of what you're doing and why you believe it is the best approach? – Peplum