Azure DevOps 2019 Server failed on executing tests with code coverage

Question

Current situation:

To enable our Azure DevOps 2019 Server (Agents) to build ASP.NET Core 3 and .Net Core 3 applications we installed Build Tools for Visual Studio 2019 (expander Tools for Visual Studio 2019) on it. It's not best practice, but Azure DevOps 2019 Server and Agents are installed on same machine.

Except collecting coder coverage everything works like expected. Applications were build and artifacts were provided. We could confirm, all artifacts after release are working like expected.
Like already mention, everything is fine except code coverage.

The application to build is a .NET Framework 4.6.2 application and was build before VS 2019 build tools installation with VS 2017 Enterprise on the agent successfully.

For testing we are using MSTest with Visual Studio Test in build pipeline, after Visual Studio build task. The test task is configured like followed:

variables:
  SolutionRootDirectory: 'MySolutionRoot'
  SettingsFilePath: ''
  BuildPlatform: 'Any CPU'
  BuildConfiguration: 'Release'

steps:
- task: VSTest@2
  displayName: 'Test Assemblies: $(BuildConfiguration)'
  inputs:
    testAssemblyVer2: |
     **/*Test?(s).dll
     !**/obj/**
    searchFolder: '$(SolutionRootDirectory)'
    runSettingsFile: '$(SettingsFilePath)'
    codeCoverageEnabled: true
    testRunTitle: 'Test run: $(SolutionRootDirectory)\*.sln'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    collectDumpOn: always
    rerunFailedTests: false

All tests were executed successfully, but the task failed with following message:

##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
##[error]Error: The process 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\Extensions\TestPlatform\vstest.console.exe' failed with exit code 1
Publishing test results to test run '11901'
Test results remaining: 260. Test run id: 11901
##[error]VsTest task failed.

After checking the whole log I found following messages:

Data collector 'Code Coverage' message: Data collector 'Code Coverage' failed to provide initialization information. Error: System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop' threw an exception. ---> Microsoft.VisualStudio.Diagnostics.Common.InvariantException: Failed to load IntelliTrace Profiler binary or failed to locate functions.

...and a few lines below:

 ---> System.ComponentModel.Win32Exception: The system cannot find the path specified
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Diagnostics.Common.Check.Throw[XT](String message, Func`1 innerEx)
   at Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop.ThrowInvariantExceptionOnZeroPtr(IntPtr ptr, String message)
   at Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop.InitInterop()
   at Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop.get_InteropInterface()
   at Microsoft.VisualStudio.Diagnostics.Logging.LoggingConfig.Publish()
   at Microsoft.VisualStudio.TraceCollector.CommonDataCollector.InitiateCollection()
   at Microsoft.VisualStudio.TraceCollector.CommonDataCollector.GetEnvironmentVariables()
   at Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector.GetEnvironmentVariables()
   at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectorInformation.SetTestExecutionEnvironmentVariables()
   at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectionManager.GetEnvironmentVariables(Boolean& unloadedAnyCollector).

Question

Does anybody have an idea what the issue could be?
I guess, maybe we have to install Agents for Visual Studio 2019 (expander Tools for Visual Studio 2019), but I'm not sure if it is the correct approach. Furthermore, I could not find any meaningfully information which installation packe we have to use - Agent or Controller.

In the past, most of the time, a fully Visual Studio Enterprise installation was done, to achieve a completely feature set for agents. But I want a more clean approach and want to install only needed packages and to avoid to use an Enterprise licence for a build agent.

Ideas, approaches and best practices are very welcome. Thank you very much for any help.

Swellfish

IIRC You need to install VS Enterprise to get code coverage. The code coverage collector is not included in the build tools. – Altimetry

Thanks for your comment. Like I have written it's one possible solution, but in my opinion it's not the clean one. I could not find articles or similar from Microsoft that are describing how to setup build agents with VS2019 build and code coverage without installing Visual Studio 2019 Enterprise. Does a VS2019 Enterprise installation needs a license on a build agent? I guess it should, but cannot find sth. about it. – Swellfish

@Swellfish I can recommand you a setup with coverlet and ReportGenerator. It works fine and is free to use. – Yaw

What about install microsoft.intellitrace.profiler.dll from opendll.com/…? See 2 solutions listed in this thread. – Dismast

Thanks for the approaches. @MarTin Yes, I already used coverlet for other projects in the past. It works well, but I think I have to update my question with my conditions. I thought Microsoft has a package like vs2019 build tools or there are possibilities to extent them to incl. code coverage like VS2019 Enterprise does. @MerlinLiang-MSFT To copy assemblies seems to be more a hack, than a best practice solution. Your 2nd URL is sadly the same like the 1st one. – Swellfish

Answer 1

applied solution

Well, after a lot of research and reading about best practices, the most practicable and most target oriented solution was to install a Visual Studio 2019 Enterprise on the build agent. It was my first approach, but asked the question for finding another solution, to avoid installing a whole VS 2019 Enterprise on the build agent.

It contains all needed assemblies and code coverage is part of Enterprise edition only.

I preferred that solution because of no build pipeline modifications are needed. Otherwise all existing build pipelines must be modified to match other suggested solutions.
If it's acceptable to edit existing pipelines, following are listed other suggested solutions.

all suggested solutions:

1. using Coverlet (Coverlet.MsBuild) by installing it via NuGet into test projects and importing its results (maybe in combination use with ReportGenerator
2. installing Test Agent/Controller and referencing its vstest.console.exe in build pipeline's MSTest task
3. copy needed binaries to relevant directories on build agent

Thank you for suggestions and your invested time.

Swellfish

Answer 2

Using Visual Studio Test Platform Installer task worked for me.

Just make sure you change vsTestVersion to toolsInstaller

- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '16.11.0'

- task: VSTest@2
  inputs:
    ...
    vsTestVersion: 'toolsInstaller'
    codeCoverageEnabled: true

(tested with version 16.11.0, released 3h ago, not sure if it was available previously)

There is one downside for me, previously I could use one variable/parameter to set both VSTest and VSBuild version as both have version parameters using same format (15.0, 16.0, latest), but the tool installer requires a specific version, so it's not straightforward how to keep that synced, however it is also not clear if we need to, perhaps not.

Lilith

Failed for me with many instances of System.Security.VerificationException: Operation could destabilize the runtime. all over the place in my tests. The use of vstest.console.exe from VS 2019 Enterprise instead of from Build Tools for VS 2019 worked for me. – Trimly