What is the best practice for "Copy Local" and with project references?

Question

I have a large c# solution file (~100 projects), and I am trying to improve build times. I think that "Copy Local" is wasteful in many cases for us, but I am wondering about best practices.

In our .sln, we have application A depending on assembly B which depends on assembly C. In our case, there are dozens of "B" and a handful of "C". Since these are all included in the .sln, we're using project references. All assemblies currently build into $(SolutionDir)/Debug (or Release).

By default, Visual Studio marks these project references as "Copy Local", which results in every "C" being copied into $(SolutionDir)/Debug once for every "B" that builds. This seems wasteful. What can go wrong if I just turn "Copy Local" off? What do other people with large systems do?

FOLLOWUP:

Lots of responses suggest breaking up the build into smaller .sln files... In the example above, I would build the foundation classes "C" first, followed by the bulk of the modules "B", and then a few applications, "A". In this model, I need to have non-project references to C from B. The problem I run into there is that "Debug" or "Release" gets baked into the hint path and I wind up building my Release builds of "B" against debug builds of "C".

For those of you that split the build up into multiple .sln files, how do you manage this problem?

Ania

You can make your Hint Path reference the Debug or Release directory by editing the project file directly. Use $(Configuration) in place of Debug or Release. E.g., <HintPath>..\output\$(Configuration)\test.dll</HintPath> This is a pain when you have a lot of references (although it shouldn't be hard for someone to write an add-in to manage this). – Incompressible

Is 'Copy Local' in Visual Studio the same as <Private>True</Private> in a csproj? – Herdic

But splitting up a .sln into smaller ones breaks VS’s automagic interdependency calculation of <ProjectReference/>s. I’ve moved from multiple smaller .slns more to a single big .sln myself just because VS causes fewer problems that way… So, maybe the followup is assuming a not-necessarily-best solution to the original question? ;-) – Selfregard

@ColonelPanic Yes. At least that is the thing that changes on disk when I change that toggle in the GUI. – Concavity

Answer 1

In a previous project I worked with one big solution with project references and bumped into a performance problem as well. The solution was three fold:

1. Always set the Copy Local property to false and enforce this via a custom msbuild step

2. Set the output directory for each project to the same directory (preferably relative to $(SolutionDir)

3. The default cs targets that get shipped with the framework calculate the set of references to be copied to the output directory of the project currently being built. Since this requires calculating a transitive closure under the 'References' relation this can become VERY costly. My workaround for this was to redefine the GetCopyToOutputDirectoryItems target in a common targets file (eg. Common.targets ) that's imported in every project after the import of the Microsoft.CSharp.targets. Resulting in every project file to look like the following:

   <Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
     <PropertyGroup>
       ... snip ...
     </ItemGroup>
     <Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />
     <Import Project="[relative path to Common.targets]" />
     <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
          Other similar extension points exist, see Microsoft.Common.targets.
     <Target Name="BeforeBuild">
     </Target>
     <Target Name="AfterBuild">
     </Target>
     -->
   </Project>
   

This reduced our build time at a given time from a couple of hours (mostly due to memory constraints), to a couple of minutes.

The redefined GetCopyToOutputDirectoryItems can be created by copying the lines 2,438–2,450 and 2,474–2,524 from C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Microsoft.Common.targets into Common.targets.

For completeness the resulting target definition then becomes:

<!-- This is a modified version of the Microsoft.Common.targets
     version of this target it does not include transitively
     referenced projects. Since this leads to enormous memory
     consumption and is not needed since we use the single
     output directory strategy.
============================================================
                    GetCopyToOutputDirectoryItems

Get all project items that may need to be transferred to the
output directory.
============================================================ -->
<Target
    Name="GetCopyToOutputDirectoryItems"
    Outputs="@(AllItemsFullPathWithTargetPath)"
    DependsOnTargets="AssignTargetPaths;_SplitProjectReferencesByFileExistence">

    <!-- Get items from this project last so that they will be copied last. -->
    <CreateItem
        Include="@(ContentWithTargetPath->'%(FullPath)')"
        Condition="'%(ContentWithTargetPath.CopyToOutputDirectory)'=='Always' or '%(ContentWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"
            >
        <Output TaskParameter="Include" ItemName="AllItemsFullPathWithTargetPath"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectoryAlways"
                Condition="'%(ContentWithTargetPath.CopyToOutputDirectory)'=='Always'"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectory"
                Condition="'%(ContentWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"/>
    </CreateItem>

    <CreateItem
        Include="@(_EmbeddedResourceWithTargetPath->'%(FullPath)')"
        Condition="'%(_EmbeddedResourceWithTargetPath.CopyToOutputDirectory)'=='Always' or '%(_EmbeddedResourceWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"
            >
        <Output TaskParameter="Include" ItemName="AllItemsFullPathWithTargetPath"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectoryAlways"
                Condition="'%(_EmbeddedResourceWithTargetPath.CopyToOutputDirectory)'=='Always'"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectory"
                Condition="'%(_EmbeddedResourceWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"/>
    </CreateItem>

    <CreateItem
        Include="@(Compile->'%(FullPath)')"
        Condition="'%(Compile.CopyToOutputDirectory)'=='Always' or '%(Compile.CopyToOutputDirectory)'=='PreserveNewest'">
        <Output TaskParameter="Include" ItemName="_CompileItemsToCopy"/>
    </CreateItem>
    <AssignTargetPath Files="@(_CompileItemsToCopy)" RootFolder="$(MSBuildProjectDirectory)">
        <Output TaskParameter="AssignedFiles" ItemName="_CompileItemsToCopyWithTargetPath" />
    </AssignTargetPath>
    <CreateItem Include="@(_CompileItemsToCopyWithTargetPath)">
        <Output TaskParameter="Include" ItemName="AllItemsFullPathWithTargetPath"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectoryAlways"
                Condition="'%(_CompileItemsToCopyWithTargetPath.CopyToOutputDirectory)'=='Always'"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectory"
                Condition="'%(_CompileItemsToCopyWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"/>
    </CreateItem>

    <CreateItem
        Include="@(_NoneWithTargetPath->'%(FullPath)')"
        Condition="'%(_NoneWithTargetPath.CopyToOutputDirectory)'=='Always' or '%(_NoneWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"
            >
        <Output TaskParameter="Include" ItemName="AllItemsFullPathWithTargetPath"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectoryAlways"
                Condition="'%(_NoneWithTargetPath.CopyToOutputDirectory)'=='Always'"/>
        <Output TaskParameter="Include" ItemName="_SourceItemsToCopyToOutputDirectory"
                Condition="'%(_NoneWithTargetPath.CopyToOutputDirectory)'=='PreserveNewest'"/>
    </CreateItem>
</Target>

With this workaround in place I found it workable to have as much as > 120 projects in one solution, this has the main benefit that the build order of the projects can still be determined by VS instead of doing that by hand by splitting up your solution.

Malissamalissia

Can you describe the changes you made and why? My eyeballs are too tired after a long day of coding to try to reverse engineer it myself :) – Benignity

How about trying to copy&paste this again - SO messed up like 99% of the tags. – Hopson

@Charlie Flowers, @Hopson edited the text to be a description could not get the xml to layout nicely. – Malissamalissia

Should the copy local be set to false only for project references or for all references ? – Testaceous

@Testaceous copy local should be set to false for at least the project references. If you add your other (non-gac) dependencies to the output path you can save some performing unnecessary copy actions. – Malissamalissia

-1 for UNCONDITIONAL recommendation to disable CopyLocal. Set CopyLocal=false can cause different issues during deployment time. See my blog post "Do NOT Change "Copy Local” project references to false, unless understand subsequences." ( geekswithblogs.net/mnf/archive/2012/12/09/…) – Equipollent

From the Microsoft.Common.targets: GetCopyToOutputDirectoryItems Get all project items that may need to be transferred to the output directory. This includes baggage items from transitively referenced projects. It would appear that this target computes full transitive closure of content items for all referenced projects; however that is not the case. – Sauna

It only collects the content items from its immediate children and not children of children. The reason this happens is that the ProjectReferenceWithConfiguration list that is consumed by _SplitProjectReferencesByFileExistence is only populated in the current project and is empty in the children. The empty list causes _MSBuildProjectReferenceExistent to be empty and terminates the recursion. So it is appears not useful. – Sauna

I use the CustomAfterMicrosoftCommonTargets property to avoid changing MSBuild targets or VisualStudio projects. For details see simple-talk.com/dotnet/.net-tools/extending-msbuild – Dight

Can you post the custom build step you use to enforce CopyLocal is set to false for every reference in every project? – Shirleneshirley

@JohannesRudolph: I'm unable to post that code since I'm no longer working on that project and don't have a copy of the check lying around. Sorry. – Malissamalissia

@BasBossink: No problem, I have found an answer here: #1682596. Would you mind linking that from this answer? – Shirleneshirley

It seems it's not possible to use the $(SolutionDir) variable in the Output Path setting. Rather, I had to use ..\bin. In any case, this was the easiest way to fix the problem for me, so thank you. – Arvind

We've been using this for many years, but it appears that it needs an update -- building under VS 2019 with dotnet build causes errors due to AssignTargetPaths not being a defined target. :( – Lareine

@ScottStafford: Sorry as you can see this answer is nearly 10 years old, I would say that a lot of the world has changed since then. – Malissamalissia

Answer 2

I'll suggest you to read Patric Smacchia's articles on that subject :

• Partitioning Your Code Base Through .NET Assemblies and Visual Studio Projects --> Should every Visual Studio project really be in its own assembly? And what does 'Copy Local=True' really mean?
• Lessons learned from the NUnit code base --> The VisualStudio Project Reference + Copy Local true option is evil!)
• Analyzing the code base of CruiseControl.NET --> Bad usage of Copy Local Reference Assembly option set to True)

CC.Net VS projects rely on the copy local reference assembly option set to true. [...] Not only this increase significantly the compilation time (x3 in the case of NUnit), but also it messes up your working environment. Last but not least, doing so introduces the risk for versioning potential problems. Btw, NDepend will emit a warning if it founds 2 assemblies in 2 different directories with the same name, but not the same content or version.

The right thing to do is to define 2 directories $RootDir$\bin\Debug and $RootDir$\bin\Release, and configure your VisualStudio projects to emit assemblies in these directories. All project references should reference assemblies in the Debug directory.

You could also read this article to help you reduce your projects number and improve your compilation time.

Southwester

I wish I could recommend Smacchia's practices with more than one upvote! Reducing the number of projects is key, not splitting the solution. – Tullusus

Answer 3

I suggest having copy local = false for almost all projects except the one that is at the top of the dependency tree. And for all the references in the one at the top set copy local = true. I see many people suggesting sharing an output directory; I think this is a horrible idea based on experience. If your startup project holds references to a dll that any other project holds a reference to you will at some point experience an access\sharing violation even if copy local = false on everything and your build will fail. This issue is very annoying and hard to track down. I completely suggest staying away from a shard output directory and instead of having the project at the top of the dependency chain write the needed assemblies to the corresponding folder. If you don't have a project at the "top," then I would suggest a post-build copy to get everything in the right place. Also, I would try and keep in mind the ease of debugging. Any exe projects I still leave copy local=true so the F5 debugging experience will work.

Spouse

I had this same idea and was hoping to find someone else who thought the same way here; however, I'm curious why this post doesn't have more upvotes. People who disagree: why do you disagree? – Cogent

No that can't happen unless the same project is building twice, why would it get overwritten\access\sharing violation if its built once and doesn't copy any files? – Aglimmer

This. If the development workflow requires building one project of the sln, while another project-exectutable of the solution is running, having everything in the same output directory will be a mess. Much better to separate executable output folders in this case. – Laurasia

Answer 4

You are correct. CopyLocal will absolutely kill your build times. If you have a large source tree then you should disable CopyLocal. Unfortunately it not as easy as it should be to disable it cleanly. I have answered this exact question about disabling CopyLocal at How do I override CopyLocal (Private) setting for references in .NET from MSBUILD. Check it out. As well as Best practices for large solutions in Visual Studio (2008).

Here is some more info on CopyLocal as I see it.

CopyLocal was implemented really to support local debugging. When you prepare your application for packaging and deployment you should build your projects to the same output folder and make sure you have all the references you need there.

I have written about how to deal with building large source trees in the article MSBuild: Best Practices For Creating Reliable Builds, Part 2.

Slipper

Answer 5

In my opinion, having a solution with 100 projects is a BIG mistake. You could probably split your solution in valid logical small units, thus simplifying both maintenance and builds.

Qktp

Bruno, please see my followup question above - if we break into smaller .sln files, how do you manage the Debug vs. Release aspect that is then baked into the hint path of my references? – Ania

I agree with this point, the solution I'm working with has ~100 projects, only a handful of which have more than 3 classes, build times are shocking, and as a result my predecessors split the solution into 3 which completely breaks 'find all references' and refactoring. The whole thing could fit in a handful of projects which would build in seconds! – Madancy

Dave, Good question. Where I work, we have build scripts that do things like build dependencies for a given solution and put the binaries somewhere where the solution-in-question can get them. These scripts are parametrized for both debug and release builds. The downside is extra time up front to build said scripts, but they can be reused across apps. This solution has worked well by my standards. – Rights

Answer 6

I am surprised no one has mentioned using hardlinks. Instead of copying the files, it creates a hardlink to the original file. This saves disk space as well as greatly speeding up build. This can enabled on the command line with the following properties:

/p:CreateHardLinksForAdditionalFilesIfPossible=true;CreateHardLinksForCopyAdditionalFilesIfPossible=true;CreateHardLinksForCopyFilesToOutputDirectoryIfPossible=true;CreateHardLinksForCopyLocalIfPossible=true;CreateHardLinksForPublishFilesIfPossible=true

You can also add this to a central import file so that all your projects can also get this benefit.

Damage

Answer 7

If you got the dependency structure defined via project references or via solution level dependencies it's safe to turn of "Copy Local" I would even say that it's a best practice todo so since that will let you use MSBuild 3.5 to run your build in parallel (via /maxcpucount) without diffrent processes tripping over each other when trying to copy referenced assemblies.

Jobie

Answer 8

our "best practise" is to avoid solutions with many projects. We have a directory named "matrix" with current versions of assemblies, and all references are from this directory. If you change some project and you can say "now the change is complete" you can copy the assembly into the "matrix" directory. So all projects that depends on this assembly will have the current(=latest) version.

If you have few projects in solution, the build process is much faster.

You can automate the "copy assembly to matrix directory" step using visual studio macros or with "menu -> tools -> external tools...".

Quantify

Answer 9

You don't need to change CopyLocal values. All you need to do is predefine a common $(OutputPath) for all projects in the solution and preset $(UseCommonOutputDirectory) to true. See this: http://blogs.msdn.com/b/kirillosenkov/archive/2015/04/04/using-a-common-intermediate-and-output-directory-for-your-solution.aspx

Enfeoff

I'm not sure this was available back in 2009, but it does seem to work nicely in 2015. Thanks! – Ania

Actual link is 404. Cached: web.archive.org/web/20150407004936/http://blogs.msdn.com/b/… – Jefferyjeffie

Answer 10

Set CopyLocal=false will reduce build time, but can cause different issues during deployment.

There are many scenarios, when you need to have Copy Local’ left to True, e.g.

• Top-level projects,
• Second-level dependencies,
• DLLs called by reflection

The possible issues described in SO questions
"When should copy-local be set to true and when should it not?",
"Error message 'Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.'"
and  aaron-stainback's answer for this question.

My experience with setting CopyLocal=false was NOT successful. See my blog post "Do NOT Change "Copy Local” project references to false, unless understand subsequences."

The time to solve the issues overweight the benefits of setting copyLocal=false.

Equipollent

Setting CopyLocal=False will sure cause some issues, but there are solutions to those. Also, you should fix the formatting of your blog, it is barely readable, and saying there that "I was warned by <random consultant> from <random company> about possible errors during deployments" is not an argument. You need to develop. – Deepdyed

@GeorgesDupéron, The time to solve the issues overweight the benefits of setting copyLocal=false. Reference to consultant is not an argument, but credit, and my blog explains what the problems are. Thanks for your feedback re. formatting, I will fix it. – Equipollent

Answer 11

I tend to build to a common directory (e.g. ..\bin), so I can create small test solutions.

Clash

Answer 12

You can try to use a folder where all assemblies that are shared between projects will be copied, then make an DEVPATH environment variable and set

<developmentMode developerInstallation="true" />

in machine.config file on each developer's workstation. The only thing you need to do is to copy any new version in your folder where DEVPATH variable points.

Also divide your solution into few smaller solutions if possible.

Feeding

Interesting... How would this work with debug vs. release builds ? – Ania

I'm not sure whether any suitable solution exists for loading debug/release assemblies through a DEVPATH, it's intended to be used for shared assemblies only, I wouldn't recommend it for making regular builds. Also be aware that assembly version and GAC are overridden when using this technique. – Feeding

Answer 13

This may not be best pratice, but this is how I work.

I noticed that Managed C++ dumps all of its binaries into $(SolutionDir)/'DebugOrRelease'. So I dumped all my C# projects there too. I also turned off the "Copy Local" of all references to projects in the solution. I had noticable build time improvement in my small 10 project solution. This solution is a mixture of C#, managed C++, native C++, C# webservice, and installer projects.

Maybe something is broken, but since this is the only way I work, I do not notice it.

It would be interesting to find out what I am breaking.

Eisteddfod

Answer 14

Usually, you only need to Copy Local if you want your project using the DLL that is in your Bin vs. what is somewhere else (the GAC, other projects, etc.)

I would tend to agree with the other folks that you should also try, if at all possible, to break up that solution.

You can also use Configuration Manager to make yourself different build configurations within that one solution that will only build given sets of projects.

It would seem odd if all 100 projects relied on one another, so you should be able to either break it up or use Configuration Manager to help yourself out.

Mingrelian

Answer 15

You can have your projects references pointing to the debug versions of the dlls. Than on your msbuild script, you can set the /p:Configuration=Release, thus you will have a release version of your application and all satellite assemblies.

Qktp

Bruno - yes, this works with Project References, which is one of the reasons we wound up with a 100 project solution in the first place. It does not work on references where I browse to the pre-built Debug releases - I wind up with a Release app built against Debug assemblies, which is a problem – Ania

Edit your project file in a text editor and use $(Configuration) in your HintPath, e.g. <HintPath>..\output\$(Configuration)\test.dll</HintPath>. – Incompressible

Answer 16

If you want to have a central place to reference a DLL using copy local false will fail without the GAC unless you do this.

http://nbaked.wordpress.com/2010/03/28/gac-alternative/

Concerted

Answer 17

If the reference is not contained within the GAC, we must set the Copy Local to true so that the application will work, if we are sure that the reference will be preinstalled in the GAC then it can be set to false.

Vocative

Answer 18

Well, I certainly don't know how the problems works out, but i had contact with a build solution that helped itself in such that all created files where put on an ramdisk with the help of symbolic links.

• c:\solution folder\bin -> ramdisk r:\solution folder\bin\

• c:\solution folder\obj -> ramdisk r:\solution folder\obj\

• You can also tell additionally the visual studio which temp directory it can use for the build.

Actually that wasn't all what it did. But it really hit my understanding of performance.
100% processor use and a huge project in under 3 Minute with all dependencies.

Kwok