.NET Core 1.0 - How to run "All tests in Solution" with xUnit command line

Question

The Getting started with xUnit.net (.NET Core / ASP.NET Core) page describes how to run tests with dotnet test command line.

It states that it requires a specific project.json, where we add xunit dependencies and test runner:

  "testRunner": "xunit",
    "dependencies": {
        "xunit": "2.1.0",
        "dotnet-test-xunit": "1.0.0-rc2-build10015"
    }

If I try calling it from the parent directory:

C:\git\Project\test [master ≡]> dotnet test
dotnet-test Error: 0 : System.InvalidOperationException: C:\git\Project\test\project.json does not exist.
   at Microsoft.DotNet.Tools.Test.TestCommand.GetProjectPath(String projectPath)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)
C:\git\Project\test [master ≡]>

Question: Is there a way to run all tests (multiple project.json) with a single dotnet test?

Silkaline

I do not think there is any. The dotnet driver would need to be smarter for that. That is one of the argumentation why they will switch back to msbuild for the .NET Core build system. – Should

For Core 2.1 you can use dotnet vstest. See #47633234 – Fouts

Answer 1

Since it's been almost a month and no answer, I'll at least share what I've been doing. (this won't be relevant once Visual Studio "15" RTM is launched because project.json is dead)

Simply using a for loop on all project.json:

Locally, from the test directory, I just run:

for /f %a in ('dir /b /s project.json ^| find /v "TestUtilities"') do dotnet test %a

Running it on all project.json except where the path has: TestUtilities

Mind that on TeamCity you need to escape % (and in scripts you need double: %%) so it goes by:

for /f %%%a in ('dir /b /s project.json ^| find /v "TestUtilities"') do dotnet test %%%a

Note the %%%. Since % in TeamCity is used for variables, the third % escapes it.

Silkaline

Doing something very similar to @Bruno, but adding for more help. Running this as a powershell script in my CI job after dotnet publish from the root of my project, multiple unit test projects are in /test: $dir = dir -dir test foreach($d in $dir){ dotnet test test\$d --no-build -xml ../TEST-$d.xml } – Dropkick

Answer 2

In case anyone looks for a Windows answer, here's oneliner in PowerShell that does the job:

dir test | % { dotnet test $_.FullName }

Disjuncture

This one did the trick on VSTS and locally Old school scripts always do the job! – Posey

Answer 3

Guys from Serilog have an example of building multiple test projects in their CI pipeline . Check out this powershell script https://github.com/serilog/serilog/blob/dev/Build.ps1#L44

Gian

Answer 4

Thanks Andrzej Lichnerowicz for initial pointer. I've been trying to integrate with AppVeyor and while this fix executed all test assemblies the build would no longer break if any tests failed.

Taking to next level I created a powershell macro, imported in to the appveyor build...

version: 1.0.{build}
install:
  - ps: Import-Module .\Appveyor.psm1
before_build:
- ps: dotnet restore
build:
  verbosity: minimal
test_script: 
- ps: Invoke-AppVeyorTest 

...and then executed the following macro:

function Invoke-AppVeyorTest 
{
    [CmdletBinding()]
    param()

    $result = "true"
    Get-ChildItem NetCoreXunit* -Recurse -Directory | % { 
        $test_path = $_.FullName
        $output = & dotnet test $test_path
        if ($output -Match ", Failed: 0, ")
        {
            Write-Output "All tests passed in $test_path"
        }
        else
        {
            Write-Output "Located failed tests in $test_path"
            $result = "false"
        }    
    }
    if ($result -eq "false")
    {
        $host.ui.WriteErrorLine("Failed tests detected.")
        exit 1
    }
}

Appveyor collates all the test results and the build once again fails if any tests failed.

Muse

Answer 5

For a cross-platform solution, you can use Node and NPM with the foreach-cli package. If you don't have a package.json in the root folder, do npm init, then:

npm install foreach-cli -D

In package.json:

"scripts : {
  ...
  "test": "foreach -g 'test/**/project.json' -x 'cd #{dir} && dotnet test'"
}

To run tests:

npm test

Wag

Answer 6

It doesn't look like this will be possible at all via the commandline, given the latest feedback from the CLI team on a recent github issue regarding the project search algorithm:

...though the team decided to move in a different direction. Specifically, we decided to have all of the commands require a path to a root artifact from which a closure is determined.

However, if you're using TFS builds there does exist an option in the dotnet build step (currently 'Preview') called "Project(s)", which accepts wildcards, so you can use the following settings to run all tests in all dotnet;

Command: 'test' Projects: '**/project.json'

Beware however, **/project.json will attempt to execute tests in all projects even if they don't have a testrunner defined, which may cause the build to fail.

Mell

This is exactly what I am trying to achieve, still, the test results look odds – Posey

test results 'look odd'? Can you elaborate on this? I have this approach working in VSTS and haven't had any issues with it so far. – Mell