No code coverage in VSTS when .runsettings file supplied

Question

We're having trouble with code coverage on our Visual Studio Online (VSTS) build definition, where no results are returned when we define a custom .runsettings file.

Locally this is working fine, however, when we update our build definition on Visual Studio Online with this file, it doesn't return any results.

My .runsettings file, I left everything in there per an example I found.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
    <TargetPlatform>x86</TargetPlatform>
    <TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
    <TreatTestAdapterErrorsAsWarnings>false</TreatTestAdapterErrorsAsWarnings>
    <CollectSourceInformation>true</CollectSourceInformation>
    <MaxCpuCount>0</MaxCpuCount>
  </RunConfiguration>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>
                <ModulePath>.*\.dll$</ModulePath>
                <ModulePath>.*\.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                <ModulePath>.*TestAdapter.*</ModulePath>
                <ModulePath>.*xamarin.*</ModulePath>
              </Exclude>
            </ModulePaths>
            <Functions>
              <Exclude>
                <Function>^Fabrikam\.UnitTest\..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
                <Attribute>^NUnit.Framework.TestFixtureAttribute$</Attribute>
                <Attribute>^Xunit.FactAttribute$</Attribute>
                <Attribute>^Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute$</Attribute>
              </Exclude>
            </Attributes>
            <Sources>
              <Exclude>
                <Source>.*\\atlmfc\\.*</Source>
                <Source>.*\\vctools\\.*</Source>
                <Source>.*\\public\\sdk\\.*</Source>
                <Source>.*\\microsoft sdks\\.*</Source>
                <Source>.*\\vc\\include\\.*</Source>
              </Exclude>
            </Sources>
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>
            <PublicKeyTokens>
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>false</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>
</RunSettings>

Update 1

Here is the output of the same run settings file on my local environment, attached to Visual Studio as the file to use:

and

What this settings file does is, to exclude the Xamarin.Forms.*.dlls from the analysis, as we should not be covering these assemblies in our project.

As you can see, it works locally, but when I use this settings file in build run in VSTS, it gives me the following:

I have tried skimming through the logs, and cannot see anything wrong with what we've done. I am in the process of setting up an on-premise TFS 2017 installation to test the same, at least I'll have access to the output directories to see what happens during the build.

I'm not familiar with this, so I'm taking stabs at any possible answer.

Mamoun

Do you specify the runsetting when do test locally? (Test=> Test Setting) Can you share the detail log on the OneDrive? (Set system.debug to true and queue build) – Vespiary

@starain-MSFT I did apply it locally, and it works. I get a successful code coverage analysis report, but the same run settings file does not work on VSTS. I've updated my original question with the update – Mamoun

Can you share the detail log on the OneDrive? – Vespiary

I can. Just the VS Test run log should be sufficient, right? – Mamoun

Yes, just VS Test log. (set system.debug to true, then queue build) – Vespiary

@starain-MSFT the log file here: 1drv.ms/f/s!AtTQ9jXAPOTxgbBzWyQEQZezL8KTJw – Mamoun

Based on the log, there is <ModulePath>.*Tests.dll</ModulePath> in Exclude section, try to remove it and try again. – Vespiary

@starain-MSFT it now has results, but it's testing only two Xamarin assemblies. My project files are completely ignored, which is way we decided to invest time in runsettings. It's just not working for us. I have two projects in the solution for which I need coverage TestApp and TestApp.Models, additionally, I have TestApp.UnitTests which is the test project. The latest test run, with the *Tests.dll removed, it only covers Xamarin.Forms.Core.dll and Xamarin.Forms.Xaml.dll which is not what I need coverage for. This is so confusing :( – Mamoun

What's the result if you run the test with CodeCoverage enabled through vstest.console.exe tool manually? – Vespiary

Can you reproduce this issue with the new projects? You can share the sample projects on the OneDrive. – Vespiary

Do you have .Net Core or .Net Standard projects? Than you have to go to project properties > Build Tab > "Advanced..." button > Debugging information Full or at least Pdb-only. I have got the same problem and searched nearly 4h until I found a solution. Hope it will solve your issue too. – Gastongastralgia

Did you find a solution to this problem? We're currently experiencing the same issue with VSTS and supplying "Other console options" with /Settings:<our runsettings file>. – Wisconsin

@Wisconsin unfortunately not. We gave up on this, and went the Sonarcloud route. – Mamoun

Answer 1

I have the following Solution on AzureDevOps OnPremise 2020, with an asp .net app. It works fine in mine scenario, where I have to transform the results to Cobertura (for sonarqube). I think for you the important task is the 'dotnet test' task (see below):

Here are the build pipeline parts:

 - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test
      arguments: '--configuration $(buildConfiguration) --no-build --logger trx --collect "XPlat Code coverage" --settings coverlet.runsettings'
      projects: '**/*Test/*.csproj'
      publishTestResults: true
  
  - task: DotNetCoreCLI@2
    displayName: Install ReportGenerator tool
    inputs:
      command: custom
      custom: tool
      arguments: update --tool-path . dotnet-reportgenerator-globaltool --ignore-failed-sources
  
  - script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.opencover.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"
    displayName: Transform coverage reports
  
  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: '$(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml'
      failIfCoverageEmpty: false

Coverate on azureDevOps:

Ebsen