Code coverage, analysis and profiling for dynamically generated code

Question

I have a demo project, which creates an assembly and uses it. I also can debug the injected code. But if I run coverage, analysis or profiling, it is counted, but I want to measure it.

Code:

CSharpCodeProvider codeProvider = new CSharpCodeProvider();
ICodeCompiler icc = codeProvider.CreateCompiler();
CompilerParameters parameters = new CompilerParameters();
parameters.GenerateExecutable = false;
parameters.GenerateInMemory = false; // debug enabled                
parameters.OutputAssembly = "DynamicCode.dll"; // if specified creates the DLL
parameters.IncludeDebugInformation = true;
CompilerResults results = icc.CompileAssemblyFromFile(parameters, "InjectedCode.cs.txt");

I create the DLL to check the generated IL code. I can debug the code in VS. But when I run coverage, the generated assembly is simply missed, if I use TEMP directory, or if I output the DLL (like above) NO FILE is included in the coverage (so not even the main assembly).

When I run profiling, I can only see the invoke (reflection), but nothing about the generated code. When I do analysis (I have some errors in the injected code, e.g not used locals, and ofc analysis for everything), no problems are reported from the injected code. The injected code:

namespace CodeInjection
{
    public static class DynConcatenateString
    {
        public static string Concatenate(string s1, string s2){
           //  System.Diagnostics.Debugger.Break(); // break here for debugger and also test comment output
            int a = 1+2+3+4+5; // complicated math
            int b = a+2;
            int c = 0;

            return s1 + " !"+b+"! " + s2;
        }
    }
}

I would like to use coverage, profiling and analysis on the generated code (mainly coverage).

Lamarckian

Answer 1

Ok code coverage in VS 2012 does not have full support for dynamically generated code.

You can run our analyze command to see that your module has been skipped for this very same reason or is it because of a missing symbol/p

Alleviator

Also skipped (was my first try after code coverage skipped). Also jetbrains cover and other tools miss. Now I am going deeper in the topic, hopefully with results in the next week, so I will send here. Please rate my question, whether you think it is useful or not. – Lamarckian

Answer 2

In your project, add a new xml file and name it dynamic.runsettings.

It should contain the following:

    <?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
<!--
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax).
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
It is considered an error to exclude all items from instrumentation as no data would be collected.
-->
              <Include>
                <!-- <ModulePath>.*\\UnitTestProject1\.dll</ModulePath> -->
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>
<!--
Additional paths to search for symbol files. Symbols must be found for modules to be instrumented.
If symbols are alongside the binaries, they are automatically picked up. Otherwise specify the here.
Note that searching for symbols increases code coverage runtime. So keep this small and local.
<SymbolSearchPaths>
  <Path>C:\Users\User\Documents\Visual Studio 11\Projects\ProjectX\bin\Debug</Path>
  <Path>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
-->
            <Functions>
              <Exclude>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
                <Function>.*::YOU_CAN_ONLY_DESIGNATE_ONE_.*</Function>
              </Exclude>
            </Functions>
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
            <Sources>
              <Exclude>
                <Source>.*\\atlmfc\\.*</Source>
                <Source>.*\\vctools\\.*</Source>
                <Source>.*\\public\\sdk\\.*</Source>
                <Source>.*\\microsoft sdks\\.*</Source>
                <Source>.*\\vc\\include\\.*</Source>
              </Exclude>
            </Sources>
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>
            <PublicKeyTokens>
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Save that. What's special about this file is the CodeCoverage\ModulePaths\Include section has its ModulePath child commented out. This will now include ANY dlls it finds in code coverage.

Now Click on Test -> Test Settings -> Select Test Settings File and specify the dynamic.runsettings file. Your code coverage will now use this custom settings file.

Click on Analyze Code Coverage and run whichever tests you'd like. You should see your DynamicCode.dll show up in your coverage results (I have obscured my workstation name below):

Note that I've added a new method to the .cs.txt file called BONK and it (correctly) shows up as 100% uncovered.

Tenebrific