xUnit tests fails when executed by Azure DevOps

Question

I'm creating a very simple test project to experiment with various Azure DevOps CI/DI features. This project contains a solution with an ASP.NET Core website project, and an xUnit unit tests project. The unit tests runs correctly on my local machine (tested with the visual studio runner and with dotnet test, but I cannot make it work with an Azure build.

The output of the unit tests step in Azure is the following :

##[section]Starting: Test Assemblies
==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 2.147.0
Author       : Microsoft Corporation
Help         : [More information](https://go.microsoft.com/fwlink/?LinkId=835764)
==============================================================================
SystemVssConnection exists true
SystemVssConnection exists true
SystemVssConnection exists true
Running tests using vstest.console.exe runner.
======================================================
Test selector : Test assemblies
Test filter criteria : null
Search folder : D:\a\1\s
VisualStudio version selected for test execution : latest
Run in parallel : false
Run in isolation : false
Path to custom adapters : null
Other console options : null
Code coverage enabled : false
Diagnostics enabled : true
SystemVssConnection exists true
Run the tests locally using vstest.console.exe
========================================================
Test selector : Test assemblies
Test assemblies : **\release\netcoreapp2.2\*test*.dll,!**\obj\**
Test filter criteria : null
Search folder : D:\a\1\s
Run settings file : D:\a\1\s
Run in parallel : false
Run in isolation : false
Path to custom adapters : null
Other console options : null
Code coverage enabled : false
Diagnostics enabled : false
Rerun failed tests: false
VisualStudio version selected for test execution : latest
========================================================
======================================================
[command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @D:\a\_temp\e481a311-390d-11e9-aa01-4f08eeab4c37.txt
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

vstest.console.exe 
"D:\a\1\s\tests\ContosoUniversity.Tests\bin\Release\netcoreapp2.2\ContosoUniversity.Tests.dll"
"D:\a\1\s\tests\ContosoUniversity.Tests\bin\Release\netcoreapp2.2\xunit.runner.visualstudio.dotnetcore.testadapter.dll"
/logger:"trx"
/TestAdapterPath:"D:\a\1\s"
Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETCoreApp,Version=v2.2 and platform X86. Following DLL(s) do not match framework/platform settings.
xunit.runner.visualstudio.dotnetcore.testadapter.dll is built for Framework 1.0 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27207.03)
[xUnit.net 00:00:04.36]   Discovering: ContosoUniversity.Tests
[xUnit.net 00:00:04.43]   Discovered:  ContosoUniversity.Tests
[xUnit.net 00:00:04.44]   Starting:    ContosoUniversity.Tests
[xUnit.net 00:00:04.93]   Finished:    ContosoUniversity.Tests
Passed   ContosoUniversity.Controllers.HomeControllerTest.Index_Renvoie_Le_Bon_Modele
Unable to find D:\a\1\s\tests\ContosoUniversity.Tests\bin\Release\netcoreapp2.2\xunit.runner.visualstudio.dotnetcore.testadapter.deps.json. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk".
Results File: D:\a\1\s\TestResults\VssAdministrator_fv-az561_2019-02-25_14_59_22.trx

Total tests: Unknown. Passed: 1. Failed: 0. Skipped: 0.
Test Run Aborted.
Test execution time: 8.5546 Seconds
##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
##[error]Error: The process 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe' failed with exit code 1
##[error]VsTest task failed.
##[section]Async Command Start: Publish test results
Publishing test results to test run '1000118'
Test results remaining: 1. Test run id: 1000118
Published Test Run : https://orkeis-proj1.visualstudio.com/MyFirstProject_Test/_TestManagement/Runs#runId=1000118&_a=runCharts
##[section]Async Command End: Publish test results
##[section]Finishing: Test Assemblies

I'm failing to understand how I'm supposed to have that missing json file. The error suggests to install Microsoft.NET.Test.Sdk, but I do have it referenced in my project file :

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>

    <IsPackable>false</IsPackable>

    <RootNamespace>ContosoUniversity</RootNamespace>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\ContosoUniversity.csproj" />
  </ItemGroup>

</Project>

So any tip about what I'm missing would be appreciated.

Reina

Answer 1

The problem was that the default file pattern to look for test assemblies (*test*.dll) also did include xUnit's core assemblies (named xunit.runner.visualstudio.dotnetcore.testadapter.dll), which confused the test system.

All I had to do to fix the issue is use a more specific pattern (like *tests.dll), or exclude all test adapter assemblies (!**/*testadapter.dll).

Update : The official xUnit documentation has been updated to help avoid this problem (https://xunit.net/docs/getting-test-results-in-azure-devops)

Reina

I was having a similar issue with NUnit where I had `***test*.dll which was looking for Test Adapters. I removed this line and now only my tests run as they should. – Shilohshim

Answer 2

Thanks to @Shtong I was able to fix my xunit pipeline build. This is my YAML file for future reference:

- task: VSTest@2
  inputs:    
    testAssemblyVer2: |
      **\bin\$(BuildConfiguration)\**\*tests*.dll         
      !**\obj\**
      !**\xunit.runner.visualstudio.testadapter.dll
      !**\xunit.runner.visualstudio.dotnetcore.testadapter.dll
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    otherConsoleOptions: '/platform:x64 /Framework:.NETCoreApp,Version=v3.1 /logger:console;verbosity="normal" '

Alrick

Answer 3

I had a similar problem, that could be solved by the solution given by Shtong.

But in my case the cause was quite different, this is why I want to share it here.

In my scenario A.Tests project references B.Tests project because some test classes inherit from B.Tests classes. This caused B.Tests.dll to be copied to A.Tests.dll location causing the same hickup in vstest.

To get rid of the error I excluded B.Tests.dll from the A.Tests.dll folder by

!**\A.Tests\**\B.Tests.dll

Alleged