A full explanation for the ResolvedFileToPublish XML element?
Asked Answered
S

1

34

By default, the ASP.NET Core SPA project templates have a section in their .csproj files like so:

<!-- Include the newly-built files in the publish output -->
<ItemGroup>
  <DistFiles Include="$(SpaRoot)dist\**; $(SpaRoot)dist-server\**" />
  <DistFiles Include="$(SpaRoot)node_modules\**" Condition="'$(BuildServerSideRenderer)' == 'true'" />
  <ResolvedFileToPublish Include="@(DistFiles->'%(FullPath)')" Exclude="@(ResolvedFileToPublish)">
    <RelativePath>%(DistFiles.Identity)</RelativePath>
    <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
  </ResolvedFileToPublish>
</ItemGroup>

I'm not finding any good documentation for this on Google, but I think this section might help me with what I want to do. I want to modify it for my own purposes but first I need to fully understand it, so could someone explain to me the following:

  • Where does SpaRoot get set?
  • What exactly does ResolvedFileToPublish do?
  • Where does DistFiles get set?
  • Where does FullPath get set?
  • What does the @(DistFiles->'%(FullPath)' "arrow notation" mean?
  • What does Exclude="@(ResolvedFileToPublish)" do?
  • What does DistFiles.Identity refer to and where does it get set?

UPDATE: This page gives some documentation on this item but not much:
https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/visual-studio-publish-profiles?view=aspnetcore-2.2#selective-file-inclusion

Sunsunbaked answered 16/8, 2019 at 14:21 Comment(5)
It is a bit hard to answer the question as it is very broad, asking deep details of the publish process together with basic msbuild language/syntax. An exhaustive answer would basically need to quote half of the documentation at MSBuild Concepts.Nodical
I'm sure it'd be possible to give some concise, bullet-point answers to the specific questions without going into reams of detail.Sunsunbaked
To answer one of my own questions: DistFiles gets set by the <DistFiles... XML elements, and apparently consists of a semicolon-delimited list of full paths to files that have been specified in the <DistFiles... elements' Include attributes. The Include attribute works in this way in this instance (pulling in files to include in the build process), because that attribute is specifically treated that way by MSBuild when attached to an element inside an ItemGroup parent.Sunsunbaked
I suggest you use this tool github.com/KirillOsenkov/MSBuildStructuredLog it's better than any documentation (that will not exist for every property) it's hierarchical, searchable, etc. twitter.com/KirillOsenkov/status/1052051332532600837 It's a must when one wants to get serious with msbuildWinnebago
thanks for asking this question...it's like MS didn't realize they could put comments in their confusing build targets for n00bs like me.Protuberant
N
76

Where does SpaRoot get set?

SpaRoot is set as a property in the project by the template. Projects contain a "static" portion; that is root-level PropertyGroup and ItemGroup elements.

Note that properties are like a global key-value dictionary (and whenever an XML node in a PropertyGroup defines a property, it overwrites existing ones with the same name). Items however are like lists; you can add (<MyItem Include="..."/>), remove (... Remove="...") and even update items (... Update="..." in static portions only, no include/remove inside targets means "update all" which you can only filter with a Condition attribute). Items are like objects, they have an "ID" which is called "Identity" and can have other properties, which are called "metadata". The "Identity" is the part that is specified in Include, which may or may not be a file name. If a file is referenced, some well-known metadata is added automatically (such as file modification dates and FullPath). Metadata can also be defined on item XML elements as either attributes (e.g. as seen in Version="1.2.3" on PackageReference items) or as child elements of the item element (e.g. RelativePath as seen above).

What exactly does ResolvedFileToPublish do?

The build is executed in the build engine by running targets that contain logic. All of the build logic that .NET projects run are controlled from MSBuild code that uses the same syntax as the project file. So by using imports or SDKs, the .csproj file itself is a build definition rather than a configuration file. Using mechanisms like BeforeTargets/AfterTargets, one can hook into the build process at specific points to execute code; in this case, the template contains a target that hooks into the publish logic.

ResolvedFileToPublish itself doesn't do anything special. The XML elements tell msbuild to add items to the ResolvedFileToPublish list based on file specifications, one of them only if the project is configured with server-side rendering (which is a property that AFAIK is also present in the static portion of the project in the template).

At a later stage during the build, targets coming from the .NET SDK use these items to compute files to copy during publish operations, tool packaging, and/or single-file publishing (3.0 feature), see Microsoft.NET.Publish.targets for the code that uses @(Microsoft.NET.Publish.targets) to access the list of items.

It is a convention that whenever properties or items are used by Microsoft-/3rd-party build logic that do not start with underscores (_), those are allowed/expected to be configured via build customizations, such as provided in the SPA templates. So we are meant to add ResolvedFileToPublish items which are considered "public API", but not _ResolvedFileToPublishAlways. By adding the built SPA files as items, we can tell the publish logic to include them during publish.

Where does DistFiles get set?

DistFiles is created by this template / logic itself. There is next to no restriction on which item or property names can be used. This could also have been named SpaDistFiles or similar. The template creates some intermediate items it can later use to create ResolvedFileToPublish items and hopes that the name doesn't conflict with any other name used in build logic.

Where does FullPath get set?

Full path is an automatic well-known property that msbuild adds to items that reference files on disk.

While an item's identity could be ClientApp\dist\myapp\index.html (or relative paths containing ..\), its FullPath metadata will be C:\path\to\proj\ClientApp\.....

What does the @(DistFiles->'%(FullPath)' "arrow notation" mean?

While properties can be accessed using the $() syntax, items are referenced using @().

When you have item MyItem with identities A and B, @(MyItem) (when evaluated to text) would be A;B. This could again be interpreted as an item specification, so passed to <OtherItem Include="@(MyItem)" />.

But the @() syntax also allows for item transformations or calling item functions (@(MyItem->Count())). Transformation is a projection of each item to another item, so in this example, @(MyItem->'X') would result in X;X since both items are transformed to the same value. To include parts of the original item, metadata values can be accessed via %(). So @(MyItem->'Hello %(Identity)') would result in Hello A;Hello B, since Identity is default metadata.

In this case, the DistFiles items which contain the path relative to the project file are transformed to reference the full path. While this is not well documented, this is needed since publishing logic expects ResolvedFileToPublish items to contain an absolute/full path since it can also be flown across project references - e.g. a library could contain publish-only assets and the consuming project needs to copy them during publish so it needs to pass the full path and not the relative path, which would not be found in the consuming project.

What does Exclude="@(ResolvedFileToPublish)" do?

An item Include="..." can be filtered to not add items that are part of the Exclude definition. In this case, the action translates to "Add the full path of DistFiles items as ResolvedFileToPublish items unless there is already an ResolvedFileToPublish item with the same identity (i.e. referring to the same file on disk)".

This is useful so as to not confuse the publish logic with duplicate items. Not sure at the moment if this would actually cause problems, but in order to be a good citizen, it is better not to cause additional file copies / file uploads (web deploy) etc.

The reason the files could already be in there is that they may have already been included by one of the default item specifications defined in the Web SDK that includes e.g. files in wwwroot or similar for publishing, depending on how your project is set up. The template just doesn't want to cause conflicts.

What does DistFiles.Identity refer to and where does it get set?

As mentioned above, items have some default metadata and Identity is one of them. In this case, the DistFiles items are created from file specifications relative to the project, so the item's identities are the project-relative paths (ClientApp\dist\...).

Since ResolvedFileToPublish items contain absolute paths, the RelativePath metadata tells the publish logic where to place the file during publish. You could also use this to rename the files or place them in subfolders.

In a verbose log / structured log, you should see that the items being added are C:\path\to\proj\ClientApp\dist\index.html with RelativePath=ClientApp\dist\index.html and CopyToPublishDirectory=PreserveNewest metadata.

Item batching

In the above code, there is a reference to metadata from within an attribute:

<RelativePath>%(DistFiles.Identity)</RelativePath>

While this tells MSBuild to set the RelativePath metadata to the source DistFiles item's Identity, this also triggers a feature called batching.

For every loose %(Item.Metadata) specification MSBuild sees (note that this only works inside targets) MSBuild groups the referenced item(s) into "batches" having the same property. It then runs the task that this is used in (in our case an intrinsic item add task) once for each batch, in which the @() notation will only yield the items from that particular batch.

When only batching on %(XYZ.Identity), this doesn't really matter and can be seen as a simple "for all".

So to be exact, the <ResolvedFileToPublish Include=... part would translate to: "For each set of DistFiles with the same Identity metadata, transform these items to their full path and, unless a ResolvedFileToPublish with this file name already exists, create a ResolvedFileToPublish item for them with the metadata RelativePath set to the DistFile item's Identity value, and the CopyToPublishDirectory metadata set to PreserveNewest."

Nodical answered 19/8, 2019 at 9:40 Comment(10)
Thanks, great answer. Just one thing (you might wanna update the answer with this): ResolvedFileToPublish is singular. Is this just a typo? It seems like it's a collection of files. Shouldn't it be ResolvedFilesToPublish?Sunsunbaked
Sadly there is no standard convention for singular/plural naming. While the items do form a collection of similar things, you could also refer to individual ones. ResolvedFileToPublish is singular as defined in the linked targets file, but other items are named in various ways - sometimes this makes sense (single .config file) and sometimes not so much (ResolvedFileToPublish)Nodical
Also added a description of batching which is used here, though not so obvious.Nodical
Thanks Martin. I've fixed a few typos; the only one I couldn't figure out was "took packaging" in the phrase "compute files to copy during publish operations, took packaging and/or single-file publishing". Could you edit it to say what you actually meant? I assume it's a typo.Sunsunbaked
Tool packaging - when you set PackAsTool to true to create .NET Core global/local tools.Nodical
Microsoft decided to kill the world. Why it is so complicated ???Gwinn
Great guide. @MartinUllrich you mentioned that "You could also use this to rename the files or place them in subfolders." when talking about RelativePath. How could I do this in a 'dynamic' way, e.g. tell the build to publish all files to a folder that's relative from '%(DistFiles.Identity)' ?Thom
I'd suggest trying <RelativePath>subfolder\%(DistFiles.RelativeDir)%(DistFiles.FileName)%(DistFiles.Extension)</...>. Maybe open a new questions if that doesn't work outNodical
This is an incredible answer, thank you! For anyone else coming here, this is the reference for properties of DistFiles learn.microsoft.com/en-us/visualstudio/msbuild/…Gery
I wanted to include an Angular app from another directory. I had to use RecursiveDir instead of RelativeDir because I use ** inside Include. <DistFiles Include="$(SpaRoot)dist\apps\my-app\**" />...<RelativePath>ClientApp\%(DistFiles.RecursiveDir)%(DistFiles.Filename)%(DistFiles.Extension)</RelativePath>...Milly

© 2022 - 2024 — McMap. All rights reserved.