Include compile file (incremental build)

Question

I have an incremental build setup as follows:

  <ItemGroup>
    <MyInput Include="$(MyInput)" />
    <UpToDateCheckInput Include="@(MyInput)" />
    <UpToDateCheckBuilt Include="$(MyOutput)" />
  </ItemGroup>

  <Target Name="MyCodeGen"
          BeforeTargets="PreBuildEvent"
          DependsOnTargets="ResolveReferences"
          Inputs="@(MyInput);$(MSBuildThisFileFullPath);$(MyCodegenExe)"
          Outputs="$(MyOutput)">
    <Exec Command="$(MyCodegenExe) $(MyCodegenParams)" />
    <ItemGroup>
      <Compile Remove="$(MyOutput)" />
      <Compile Include="$(MyOutput)" />
    </ItemGroup>
  </Target>

The target produces a single .cs file $(MyOutput) that needs to be included and compiled in the current project. Note in the target I'm first removing then including this file.

1. On first build, the target runs as expected.
2. On second build, the target does not run, as expected. But MSBuild still compiles because $(MyOutput) "has been modified since the last up-to-date check".
3. On third build, everything is up-to-date.

How can I get an "everything is up-to-date" result on the second build?

Thema

Is it necessary for you to remove and then include the $(MyOutput) file? If you delete this “Remove-Include” item group, does it work? – Chavez

@Chavez if I don't include then $(MyOutput) is not compiled on the first build. If I don't remove then I get a warning about $(MyOutput) being included twice on the second build. – Thema

If your target always runs anyway, you could move the ItemGroup out of the target and just always Include the output. – Xanthous

@Xanthous it doesn't run always, only if relevant files changed. – Thema

Sorry meant to say 'if the file is always present it can be added to the Compile ItemGroup unconditionally' Which is the case, I assume? If so that should work as expected. At least I have several projects which generate files in the prebuild stage and that works as it should. – Xanthous

@Xanthous it's not always present. I don't add generated files to source control. – Thema

Sure, but is it always present for the build i.e. does it get created when it does not exist? As long as that is the case you can put it in Compile without problems, just that VS or similar might indicate it as not present on first build after checkout, but that is a non-issue. – Xanthous

@Xanthous it gets created but not included in the compile items, unless I add it inside the target, like I have it now. – Thema

@Xanthous Wait, I found that you can move the "remove-include" item group outside the target. But the original issue is still there: on the second build $(MyOutput) is treated as modified by FastUpToDate. – Thema

@MaxToro do you generate the file into the $(IntermediateOutputDirectory) or somewhere where the glob patterns would pick the .cs file up (and show them in VS solution explorer)? If so, there's little that can be done (there is a _GenerateCompileDependencyCache private target that captures all Compile items in the project, this will change during first and second build) – Toon

@MartinUllrich The output file goes to the root of the project, so yes it is picked up by the default glob. – Thema

Are you sure that the second build, MSBuild still compiles is caused by the last up-to-date check? Did some tests, the CoreCompile could be skipped successfully on second build on my side but it seems the issue is not that easy, as I saw 1 succeeded, 0 up-to-date instead of 0 succeeded, 1 up-to-date. Maybe it’s a potential issue, I’m not sure. What about also reporting/asking this issue in Microsoft Developer Community? – Chavez

@Chavez Did you set Proj and Solution > SDK-Style Proj >Loging Level = Minimal – Thema

Answer 1

Including the file unconditionally should work. For example with these lines in a default C# project template right after the Microsoft.CSharp.targets Import:

<ItemGroup>
  <MyInput Include="in.txt" />
</ItemGroup>
<PropertyGroup>
  <MyOutput>Generated.cs</MyOutput>
</PropertyGroup>
<Target Name="MyCodeGen" BeforeTargets="PreBuildEvent" DependsOnTargets="ResolveReferences"
        Inputs="@(MyInput);$(MSBuildThisFileFullPath)"
        Outputs="$(MyOutput)">
  <ReadLinesFromFile File="@(MyInput)">
    <Output TaskParameter="Lines" ItemName="MyInputlines"/>
  </ReadLinesFromFile>
  <Message Text="CODEGEN lines @(MyInputlines)" />
  <WriteLinesToFile File="$(MyOutput)" Lines="@(MyInputlines)" Overwrite="True" />
</Target>
<ItemGroup>
  <Compile Include="$(MyOutput)"/>
</ItemGroup>

The results of running msbuild /v:d on the commandline are (showing only output for MyCodeGen Target):

Output of first run (Generated.cs does not exist yet):

Target "MyCodeGen" in project "my.csproj" (target "PreBuildEvent" depends on it):
Building target "MyCodeGen" completely.
Output file "Generated.cs" does not exist.
Using "ReadLinesFromFile" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "ReadLinesFromFile"
Done executing task "ReadLinesFromFile".
Task "Message"
  CODEGEN lines using System;
Done executing task "Message".
Using "WriteLinesToFile" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "WriteLinesToFile"
Done executing task "WriteLinesToFile".
Done building target "MyCodeGen" in project "my.csproj".

Output of second/third/... run:

Target "MyCodeGen" in project "my.csproj" (target "PreBuildEvent" depends on it):
Skipping target "MyCodeGen" because all output files are up-to-date with respect to the input files.
Input files: in.txt;my.csproj
Output files: Generated.cs
Done building target "MyCodeGen" in project "my.csproj".

Output of build after modifying in.txt:

Target "MyCodeGen" in project "my.csproj" (target "PreBuildEvent" depends on it):
Building target "MyCodeGen" completely.
Input file "in.txt" is newer than output file "Generated.cs".
Using "ReadLinesFromFile" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "ReadLinesFromFile"
Done executing task "ReadLinesFromFile".
Task "Message"
  CODEGEN lines using System.Linq;
Done executing task "Message".
Using "WriteLinesToFile" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "WriteLinesToFile"
Done executing task "WriteLinesToFile".
Done building target "MyCodeGen" in project "my.csproj".

Output of runs after this:

Target "MyCodeGen" in project "my.csproj" (target "PreBuildEvent" depends on it):
Skipping target "MyCodeGen" because all output files are up-to-date with respect to the input files.
Input files: in.txt;my.csproj
Output files: Generated.cs
Done building target "MyCodeGen" in project "my.csproj".

If this doesn't work for you, it could be for instance that the process you use to generate the file doesn't play well with msbuild in that it doesn't update the timestamp correctly. E.g. just copying an existing file which is older than the project file would lead to Generated.cs having a timestamp older than the $(MSBuildThisFileFullPath) input so the target would just always run.

Xanthous

Suggest adding the MyOutput to DefaultItemExcludes so there isn't a duplicate item - that could confuse the project system in some weird cases when someone manually edits it using the property page. – Toon

I don't have an issue with skipping MyCodeGen on the second build. The issue is that CoreCompile is not skipped. But I found that the issue occurs only in VS, not when ivoking MSBuild from the command line. – Thema

@MartinUllrich DefaultItemExcludes works as an alternative to removing the output file. – Thema

I cannot reproduce that; build in VS when Generated.cs does not exist invokes MyCodeGen/CoreCompile/... second build in VS just outputs ========== Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped ========== i.e. actual build didn't kick off. Did you use the code as above? Does your code generation do anything different than the dummy implementation shown? – Xanthous

Are you using <Project Sdk="Microsoft.NET.Sdk">? – Thema

Tried that as well and the result is similar, second build skips CoreCompile. Also with EnableDefaultCompileItems=True i.e. only using the MyCodeGen target but without adding Generated.cs, because it gets picked up automatically anyway. However if you want an answer for skd-style projects YMVV and you should mention that explicitly in the question and more important: post a minimal project since there is no (AFAIK) 'standard' way of generating such project files using VS2019 for instance so we don't really know what you're trying. – Xanthous

@Xanthous If you don't add the generated file it's not compiled on the first build. You can test this by adding another file that references a type from the generated file. – Thema

Right. Then add it conditionally like <Compile Include="$(MyOutput)" Condition="!@(Compile->AnyHaveMetadataValue('Identity', $(MyOutput)))"/> – Xanthous