dotnet cli - specified .runsettings file not used in code coverage run

Question

I have a solution containing two dotnet core 2.1 projects (c#).

1. The first is a console application

2. The seconds is a test project with unit tests

I generate code coverage stats about project 1 when executing tests in project 2 using this command:

dotnet test C:\tempDir\SampleApp\Tests\SampleApp.Tests.csproj 
/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura 
/p:CoverletOutput=C:\tempDir\Coverage\coverage 
/p:settings=CodeCoverage.runsettings --filter Category=Unit --logger trx 
--results-directory C:\tempDir\output

You can see here I specify CodeCoverage.runsettings as the settings parameter - /p:settings=CodeCoverage.runsettings. In my run settings file, I've asked that Program.cs and Startup.cs are excluded from coverage, but they are still included in the output coverage.cobertura.xml file.

Extract from output report below:

<classes>
    <class name="SampleApp.Startup" filename="SampleApp\Startup.cs" line-rate="1" branch-rate="0" complexity="2">
      <methods>
        <method name="ConfigureAppConfiguration" signature="(Microsoft.Extensions.Configuration.IConfigurationBuilder)" line-rate="1" branch-rate="0">
          <lines>
            <line number="18" hits="1" branch="False" />
            <line number="19" hits="1" branch="False" />
            <line number="20" hits="1" branch="False" />
          </lines>
        </method>
        <method name="ConfigureLogging" signature="(Microsoft.Extensions.Configuration.IConfiguration,Microsoft.Extensions.Logging.ILoggingBuilder)" line-rate="1" branch-rate="0">
          <lines>
            <line number="23" hits="1" branch="False" />
            <line number="24" hits="1" branch="False" />
            <line number="25" hits="1" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="27" hits="1" branch="False" />
          </lines>
        </method>
      </methods>
      <lines>
        <line number="18" hits="1" branch="False" />
        <line number="19" hits="1" branch="False" />
        <line number="20" hits="1" branch="False" />
        <line number="23" hits="1" branch="False" />
        <line number="24" hits="1" branch="False" />
        <line number="25" hits="1" branch="False" />
        <line number="26" hits="1" branch="False" />
        <line number="27" hits="1" branch="False" />
      </lines>
    </class>
</classes>

I'm wondering what I've done wrong in my runsettings file? (contents of file below)

<?xml version="1.0" encoding="utf-8"?>

<RunSettings>
    <!-- Configurations for data collectors -->
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <CodeCoverage>

                        <ModulePaths>
                            <Include>
                                <ModulePath>.*dll$</ModulePath>
                            </Include>
                            <Exclude>
                                <ModulePath>.*microsoft.*</ModulePath>
                                <ModulePath>.*moq.*</ModulePath>
                                <ModulePath>.*polly.*</ModulePath>
                                <ModulePath>.*fluentassertions.*</ModulePath>
                                <ModulePath>.*newtonsoft.*</ModulePath>
                                <ModulePath>.*SampleApp.Tests.*</ModulePath>
                                <ModulePath>.*\\[^\\]*DocumentManagement[^\\]*\.dll</ModulePath>
                            </Exclude>
                        </ModulePaths>

                        <Functions>
                            <Exclude>
                                <Function>.*\.Program\..*</Function>
                                <Function>.*\.Startup\..*</Function>
                                <Function>.*\.SomeOtherClass\..*</Function>
                            </Exclude>
                        </Functions>

                        <Attributes>
                            <Exclude>
                                <Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
                                <Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
                                <Attribute>^System\.Runtime\.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                                <Attribute>^System\.CodeDom\.Compiler.GeneratedCodeAttribute$</Attribute>
                                <Attribute>^System\.Diagnostics\.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
                            </Exclude>
                        </Attributes>

                        <!-- We recommend you do not change the following values: -->
                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <CollectAspDotNet>False</CollectAspDotNet>

                    </CodeCoverage>
                </Configuration>
            </DataCollector>

        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

Not sure why this section is still here in this output report, when I specified it being skipped in the runsettings file.

NOTE: I'm trying to avoid littering my code with the [ExcludeFromCodeCoverage] attribute and I don't want to have to start adding /p:ExcludeByFile=Program.cs or /p:ExcludeByFile=Startup.cs to my test command in builds, hence using the runsettings file.

Wort

Did you try removing dll from the include section ? – Hammers

But I want the dll, just not two classes within the dll... – Wort

Any update? In java and nodejs is so easy – Arathorn

Answer 1

You cannot exclude classes using runsettings file by just providing the class name.

The Function element from run setting matches the full name of a a function / method like

YourNamespace.YourClass.Method(parameters);

There are only below settings which are possible from the documentation :

Other ways to include or exclude elements ModulePath - matches assemblies specified by assembly file path.

CompanyName - matches assemblies by the Company attribute.

PublicKeyToken - matches signed assemblies by the public key token.

Source - matches elements by the path name of the source file in which they are defined.

Attribute - matches elements to which a particular attribute is attached. Specify the full name of the attribute, and include "Attribute" at the end of the name.

Function - matches procedures, functions, or methods by fully qualified name. To match a function name, the regular expression must match the fully qualified name of the function, including namespace, class name, method name, and parameter list.

What options you have:

Option 1: Using Starts with OR Using method name

<Functions>
  <Exclude>

    <!-- Exclude all methods in SampleApp.Program : -->
    <Function>^SampleApp\.Program\..*</Function>

    <!-- Exclude all methods named Main: -->
    <Function>.*\.Main\(.*</Function>
  </Exclude>
</Functions>

In first Function, please note that your namespace with class name is specified and it is starting with ^ character.

In second function element, please note that it is checking method name by checking if a string ends with opening parenthesis '('.

Option 2: You can use attributes on classes and exclude them from the runsettings file.

This is similar to ExcludeFromCodeCoverage attribute.

Please note the complete runsettings file at the end of this documentation page.

Hammers

Can you explain what the ^ and (.* characters are doing in the regular expression examples? The example you've given is for include - do you have an equivalent for exclude? An example function might be SampleApp.Program.Main (which is the main method of my app) - how could I exclude that? Thanks for your help!! – Wort

^ sinigies that the fully qualified name should start with this string. ( is required if you want to specify exclusion of specific methods. I hav eupdated exclude example. – Hammers

Still doesn't work for me when I use this in the runsettings file. I got it working with the following from the command line (which isn't elegant). This does work from command line: dotnet test C:\tempDir\SampleApp\Tests\SampleApp.Tests.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=C:\tempDir\Coverage\coverage /p:Exclude="[*]*Startup%2c[*]*Program" /p:ExcludeByAttribute="Obsolete%2cExcludeFromCodeCoverage" --filter "Category=Unit" – Wort

@RobMcCabe, Please add your tip as answer. It works! /p:Exclude="[*]*Startup%2c[*]*Program" – Arathorn