C# Incremental Generator - How I can read additional files? AdditionalTextsProvider not working as expected

Question

I am trying to get some values from the appsettings.json. But whatever I try with the AdditionalTextsProvider doesn't work. Here is my code

IncrementalValuesProvider<AdditionalText> textFiles = context.AdditionalTextsProvider.Where(static file => file.Path.Contains("appsettings.json")); // tried many things here, like EndsWith(".json") etc..
IncrementalValuesProvider<(string name, string content)> namesAndContents = textFiles.Select((text, cancellationToken) => (name: Path.GetFileNameWithoutExtension(text.Path), content: text.GetText(cancellationToken)!.ToString()));

context.RegisterSourceOutput(namesAndContents, (spc, nameAndContent) =>
    {
         nameAndContent.content; //always empty 
         nameAndContent.name; //always empty 
    });

From the other hand, when I implement the ISourceGenerator (same solution, same projects) this line of code just works!

var file = context.AdditionalFiles.FirstOrDefault(x => x.Path.Contains("appsettings.json"));

The project which is referencing the code generator:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Dapper" Version="2.0.123" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.4" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="6.0.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="6.0.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="6.0.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Scrutor" Version="4.1.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.3.0" />
  </ItemGroup>
  <ItemGroup>
        <ProjectReference Include="..\myproject\myproject.csproj" />
    <ProjectReference Include="..\myproject.EFCore\myproject.EFCore.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>

  <ItemGroup>
    <AdditionalFiles Include="appsettings.json" />
  </ItemGroup>

</Project>

Code generator project :

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <LangVersion>latest</LangVersion>    
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild> <!-- Generates a package at build -->
    <IncludeBuildOutput>false</IncludeBuildOutput> <!-- Do not include the generator as a lib dependency -->
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3" PrivateAssets="all" />
  </ItemGroup>
  <ItemGroup>
    <!-- Generator dependencies -->
    <PackageReference Include="Newtonsoft.Json" Version="13.0.1" GeneratePathProperty="true" PrivateAssets="all" />
  </ItemGroup>
 <PropertyGroup>   <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
  </PropertyGroup>
  <Target Name="GetDependencyTargetPaths">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="$(PKGNewtonsoft_Json)\lib\netstandard2.0\Newtonsoft.Json.dll" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>
    <ItemGroup>
    <!-- Package the generator in the analyzer directory of the nuget package -->
    <None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
    <!-- Package the props file -->
  </ItemGroup>
</Project>

Sakmar

I had mine fail when i tried to branch off the AdditionalTextsProvider multiple times and in general i have to say, I only got simple Incremental SGs to work, ok I didn't try since a long time, since I have by now implemented my own cached SG pipeline and I'm not dealing with those crappy ISGs anymore till MS seriously fixes them and their interface. – Creatine

Yea.. and in general I hate the IIncrementalGenerator API. It just feels weird, its ugly.. – Sakmar

Answer 1

If you combine the additional text provider with the compilation it works for me (inspired by this great post):

public void Initialize(IncrementalGeneratorInitializationContext context)
{
    var files = context.AdditionalTextsProvider
        .Where(a => a.Path.EndsWith("appsettings.json"))
        .Select((a, c) => (Path.GetFileNameWithoutExtension(a.Path), a.GetText(c)!.ToString()));

    var compilationAndFiles = context.CompilationProvider.Combine(files.Collect());
            
    context.RegisterSourceOutput(compilationAndFiles, (productionContext, sourceContext) => Generate(productionContext, sourceContext));
}

void Generate(SourceProductionContext context, (Compilation compilation, ImmutableArray<(string, string)> files) compilationAndFiles)
{
    //emit generated files...
}

Update:

Tested again with Visual Studio 17.4. and combination with compilation seems not to be necessary anymore. Generator is called as expected when appsettings.json changes with only AdditionalTextsProvider registered.

Philander

If you combine the compilation with the selected provider the generator is run on all changes and no cache hits or selection are done. This would be the same as using old ISourceGenerator. See github.com/dotnet/roslyn/blob/main/docs/features/… – Eglanteen

That was the solution for me! Reading the files on Initialize works! – Sakmar

Just make sure that the <ItemGroup> <AdditionalFiles Include="appsettings.json" /> </ItemGroup> is in the project that REFERENCES the source generation project, NOT the source generation project itself. – Deckard

Answer 2

I have tried replicate but everything works for me.

I define my source generator as this

namespace ConsoleAppSourceGenerator
{
    [Generator]
    public class TxtGenerator : IIncrementalGenerator
    {
        private static int counter;

        public void Initialize(IncrementalGeneratorInitializationContext initContext)
        {
            IncrementalValuesProvider<AdditionalText> textFiles = initContext.AdditionalTextsProvider
                .Where(static file => file.Path.EndsWith(".txt"));

            IncrementalValuesProvider<(string name, string content)> namesAndContents = textFiles
                .Select((text, cancellationToken) => (name: Path.GetFileNameWithoutExtension(text.Path), content: text.GetText(cancellationToken)!.ToString()));

            initContext.RegisterSourceOutput(namesAndContents, (spc, nameAndContent) =>
            {
                spc.AddSource($"TxtFile.{nameAndContent.name}.g.cs", $@"
// Counter {Interlocked.Increment(ref counter)}
public static partial class TxtFile
{{
    public const string {nameAndContent.name} = ""{nameAndContent.content}"";
}}");
            });
        }
    }
}

and source generator project as this

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
      <Nullable>enable</Nullable>
      <LangVersion>Latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" />
  </ItemGroup>
</Project>

Then in another project, where I have made a dummy foo.txt file I have

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
      <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
      <CompilerGeneratedFilesOutputPath>Generated</CompilerGeneratedFilesOutputPath>
  </PropertyGroup>

    <ItemGroup>
        <Compile Remove="$(CompilerGeneratedFilesOutputPath)/**/*.cs" />
    </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\ConsoleAppSourceGenerator\ConsoleAppSourceGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>
    
    <ItemGroup>
        <AdditionalFiles Include="foo.txt"/>
    </ItemGroup>

</Project>

If I through Visual Studio look at Dependencies->Analyzers->ConsoleAppSourceGenerator>TxtFile.foo.g.cs then it correctly increase the counter on every change I to in the file (see step 9).

I store generated files in path ConsoleApp/Generated/ConsoleAppSourceGenerator/ConsoleAppSourceGenerator.TxtGenerator and whenever I rebuild the project it correctly updates the files.

Have you remembered to add the [Generator] attribute to the IIncrementalGenerator?

Could you test this out and validates this doesn't either work for you?

Eglanteen

Answer 3

"AdditionalFiles" in your consuming project file is the key to ensuring these files are captured by your source/incremental generator.

Found out that using a "<AdditionalFiles Include='...' />" is not necessary in some projects such as ASPNET Core projects for Source/IncrementalGenerators. The .cshtml files are included as additional files by default.

Also, file globbing patterns work with the "Include" parameter.

Keneth