Build script to copy files from various source folder to various destination folder

Question

The basic requirement is to copy the various files and folders from different solution/project directories to the single build_output folder(/subfolders).

Currently, I am doing this operation using the Robocopy commands. The only issue is my script is too long just using multiple Robocopy commands.

<Copy SourceFiles="$(Docs)\Manual.pdf" DestinationFolder="$(BuildPath)\Help"/>
<RoboCopy Source="$(Web1)" Destination="$(BuildPath)" Files="*.aspx" Options="/E"/>
<RoboCopy Source="$(Web1)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<RoboCopy Source="$(Web2)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<!--- 100s of such RoboCopy & Copy commands (note that in last two commands i need to copy from different sources to same destination -->

1. How this job is implemented in real enterprise applications, so that the build script is concise and clear.
2. Is my thinking below is the way to approach the solution. If yes, can anybody provide me sample steps using MSBuild or CommandScript easily. (free to use any MSBuild extensions)
   • Define the mapping of the all source folders, file types (can be xyz.png/.png/.*) and the destination path.
   • Copy the files (Robocopy) using the above mentioned mappings using a single target or task)
3. Is there any other better way to do this problem?

Insights/Solution ???

Predominance

Answer 1

I do exactly this sort of thing to stage build output for harvesting by the installer build. I have a custom targets file for consistent processing and have some msbuild property files with the item groups describing that needs to be done.

  <ItemGroup Label="AcmeComponent1Payload">
    <FileToHarvest Include="$(SourceRoot)AcmeProjects\ServerManager\$(Configuration)\**\*;
                            $(SourceRoot)Library\SQLServerCompact\**\*;
                            $(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.*"
                   Exclude="$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\IntegrationTests.*;
                            $(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Practices.*.xml;
                            $(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\obj\**\*;
                            $(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.VisualStudio.*;
                            $(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Web.*;
                            $(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.xml">
      <Group>AcmeServerManager</Group>
      <SubDir>Utilities\</SubDir>
    </FileToHarvest>
  </ItemGroup>

The custom targets file has the functionality to process it.

  <Target Name="CopyFiles">
    <Copy Condition="@(FileToHarvest)!=''"
          SourceFiles="@(FileToHarvest)"
          DestinationFiles="@(FileToHarvest->'$(OutputPath)\%(Group)\%(SubDir)%(RecursiveDir)%(Filename)%(Extension)')"
          OverwriteReadOnlyFiles="true"
          SkipUnchangedFiles="true" />
  </Target>

You can make the properties file as simple or as complicated as you like. I use multiple ones and import them into the project file using wildcards.

Alishiaalisia

You can of course implement your own copy step to use robocopy instead of the built in copy task. If you use exec, remember that robocopy succeeds on exit codes 0-7 so remember to set it to ignore the exit code and process the value yourself. – Alishiaalisia

Answer 2

Thanks @daughey, I got my first part working, where I need to copy from different sources to the same destination.

<!--Declare an ItemGroup that points to source Locations-->
  <ItemGroup>
    <ItemToCopy Include="$(Web1)\Audit"/>
    <ItemToCopy Include="$(Utilities)\Service"/>
    <ItemToCopy Include="$(Web1)\NET"/>
  </ItemGroup>

  <!--Declare an ItemGroup that points to destination Locations-->
  <ItemGroup>
    <DestLocations Include="$(BuildPath)" />
  </ItemGroup>

  <Target Name="CopyFiles">
    <!-- Run the copy command to copy the item to your dest locations-->
    <!-- The % sign says to use Batching. So Copy will be run for each unique source ItemToCopy(s) in the DestLocation.-->
    <RemoveDir Directories="$(BuildPath)"/>
    <Message Importance="high" Text="Deploy folder is $(BuildPath)"/>
    <RoboCopy Source="%(ItemToCopy.FullPath)" Destination="$(BuildPath)" Files="*.dll"/>
  </Target>

After struggling with Item Metadata for Task Batching, the second part is also working great. Scenario: Copy files from list of source directories to output directory on their respective sub-folders

• $(Web1)\Audit*.dll => $(BuildPath)\Audit*.dll

• $(Utilities)\Service*.jpg => $(BuildPath)\Utilities\Service*.jpg

Solution

<!--Modify an ItemGroup with metadata-->
<ItemGroup>
    <ItemToCopy Include="$(Web1)\Audit">
        <ToPath>$(BuildPath)\Audit</ToPath>
        <FileType>*.dll</FileType>
    </ItemToCopy>
    <ItemToCopy Include="$(Utilities)\Service">
        <ToPath>$(BuildPath)\Utilities\Service</ToPath>
        <FileType>*.jpg;*.bmp</FileType>
    </ItemToCopy>
</ItemGroup>

<Target Name="CopyBatch">
    <RoboCopy Source="%(ItemToCopy.Identity)" Destination="%(ItemToCopy.ToPath)" Files="%(ItemToCopy.Filetype)"/>
</Target>

Predominance

I recommend using %(ItemToCopy.FullPath) instead of %(ItemToCopy.Identity). The identity corresponds to what you entered in the Include statement and usually works. There are some scenarios where it may fail. FullPath is always the correct value. learn.microsoft.com/en-us/visualstudio/msbuild/… – Alishiaalisia