How to speed up aspnet_compiler.exe?

Question

We have Asp.Net Mvc application for which aspnet_compiler.exe takes more than 5 minutes to run.

The Asp.Net Mvc application is question depends on ~100 smaller projects that all contribute to it different static views, Javascript files, etc... by copying them from their own project folders. These small projects are not web applications in themselves, but they contain web content in order to distribute it between the different projects.

At the end, everything is consolidated under a single web application. And then we run aspnet_compiler.exe, which takes more than 5 minutes. Ouch.

The code targets .Net Framework 4.7.2 and the web applications are not SDK style.

We obviously doing something wrong. How can we reduce this time?

EDIT 1

The whole solution takes ~14.5 minutes to build from scratch using msbuild with /m:12. The msbuild node utilization is not good. According to the detailed build summary it is:

============================== Node Utilization (IDs represent configurations) ====================================================
Timestamp:            1       2       3       4       5       6       7       8       9       10      11      12       Duration   Cumulative
...
Utilization:          22.8    11.7    17.2    7.2     52.4    16.6    12.8    13.3    13.1    29.8    34.3    14.9     Average Utilization: 20.5104451261997

It is my belief that the poor utilization is due to aspnet_compiler being called relatively late in the build at a point where all the remaining projects depend on the main web application to finish building. I have captured the project build events as chrome traces (inspired by https://github.com/rainersigwald/TraceEventLogger) here is the bird view picture:

What looks like a long bridge in the middle is the build event for the main web application. It takes 8:55 minutes to build of which the AspNetCompiler task takes 5:15 minutes.

And so I am trying to understand why it takes so long? I am also exploring other possibilities, like arrange it as a standalone project so that msbuild could queue it on parallel with the projects depending on the main web application. But the main question remains - what can be done about the Asp.Net view precompilation to make it run faster.

Unfortunately, it is a block box for me. I do not really know anything about how it works and what are the ways to optimize it.

Ardellearden

The only thing we as a reader can count here are "5 minutes" and "gazillion". We could give generic advice like "buy an SSD" or "reduce the number of files to a brazillion", but nothing more, really. – Furunculosis

Right. What info should I provide to make it better? – Ardellearden

Maybe track the number of files it's handling, and if it's really a lot, accept it as a fact of life, improve the hardware or try to reduce their number? – Furunculosis

I understand how to trace the regular msbuild, but aspnet_compiler is a black box for me. It does not produce a binary log, the information on the web is scarce or I am looking in the wrong places. As for the file count - are you looking at aspx and cshtml only or other kinds of files as well? – Ardellearden

so when you go build project it is 5 minutes, or is it taking 5 minutes to launch the site say with f5? So keep the two "issues" separate. So a built project, or a re-build all is 5 minutes? And was it always this slow, or all of a sudden? Try pulling out your network plug, and do a build - does that help? – Sabayon

I have provided some details. Is it any better now? – Ardellearden

Do you perhaps have this setting in the MVC csproj file: <MvcBuildViews>true</MvcBuildViews>? I know it serves a useful purpose, but you may not need it to be enabled always, and it really slows down the build. If present, try setting it to false, this may help especially during the development cycle with frequent builds in order to run the application after a small change. – Tommyetommyrot

Developers are already advised to set it to false locally. But it is undefined on PR/CI builds. – Ardellearden

Answer 1

aspnet_compiler is slow and does not allow/support parallel compilation internally. It was never updated for performance, or for basic parallel compilation, and is "legacy junk" at this point. It is also not possible to run multiple instances concurrently on the same site. As aspnet_compiler generates many temp files, using a faster drive may help: however, even with a fastest SSD, the CPU is significantly under-utilized.

Using Roslyn can speed up compilation >2x over the .NET Framework provided compiler, and is the only way I've found to improve performance of 'the core' compilation performance. It also improves site warmup speed as a result of faster compilation.

Anecdotal performance when using Roslyn¹,

• MSBuild compiles 'two hundred assemblies' in ~5 minutes
• aspnet_compiler compiles website in ~40m (was ~100m before Roslyn)

I strong recommend using Rosyln, even if this doesn't make the site compilation "fast"; it also allows access to C#7 features from views and ASPX files. Ensure that Rosyln is correctly using the VBCSCompiler.exe service model.

It is more efficient, assuming there are no compilation errors to dig through, to compile an IIS site once instead of once for each application in the site.

For CI/CD pipelines, use release builds if possible. Having both release builds and release web.config settings (ie. debug="false") results in faster compilation. Results appear to be more mixed when using release web.config settings with debug builds. YMMV.

Anecdotal performance when using release vs debug builds,

• CI/CD server build for debug in ~35m
• CI/CD server build for release in ~25m

For local development, use in-place compilation if possible. In-place compilation allows the option for incremental compilation, where only modified files are recompiled by default.

Anecdotal performance when using an in-place compilation for local development,

• first compilation in-place in ~40m
• second compilation with no files changed in ~10s

An incremental in-place compilation can result in some errors, such as if ABI in libraries breaks; a full recompilation (-c option) will clear the issue. CI/CD builds should probably always perform a full compilation.

Update "JScript" files to C#/VB.NET

Neither Roslyn nor CodeDom apply for this legacy language, and appearance of such pages must run through some old compiler process which is much, much, slower.

Unfortunately, aspnet_compiler also lacks sufficient output generation to determine which part(s) of a site are taking the most time to compiler.

---

¹The website I am dealing with currently has 189 compiled assemblies, 97 .INC and .ASP classic files, 1580 .ASPX files (mix of C# WebForms and 'script'), and 1221 .CSHTML files. Final .COMPILED file count in IIS is 5788. Performance improvements will likely vary based on a number of factors including types of files and dependency graphs.

---

Cliff notes on setting up Roslyn for aspnet_compiler:

• Download the Rosyln CodeDom package (Microsoft.CodeDom.Providers.DotNetCompilerPlatform). This includes both the CodeDom and a build of Roslyn.

• Ensure the Roslyn directory in the package ends up in bin/roslyn, however such fits into the build system. Here there is a manual pre-build copy step. Likewise, the CodeDom bridge assembly must be in bin/.

• Update the system.codedom/compilers section in the web.config file as shown in the example. If not using VB.NET, that compiler entry can be removed along with the VB-specific Roslyn analyzers. (Note that below I've set /langversion:7 for C#7 features, the highest set officially supported in any .NET Framework target.)

  <system.codedom>
      <compilers>
        <compiler language="c#;cs;csharp" extension=".cs"
          type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.CSharpCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
          warningLevel="4" compilerOptions="/langversion:7 /nowarn:1659;1699;1701"/>
        <compiler language="vb;vbs;visualbasic;vbscript" extension=".vb"
          type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.VBCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
          warningLevel="4" compilerOptions="/langversion:14 /nowarn:41008 /define:_MYTYPE=\&quot;Web\&quot; /optionInfer+"/>
      </compilers>
  </system.codedom>
  

Verifying that Roslyn is invoked correctly:

Kick off a site build. During compilation there should be many csc.exe instances launched. However, these executable instances should not take up much memory. Instead, there should be another executable called VBCSCompiler.exe that runs from the bin/roslyn directory (Visual Studio may spawn it's own Roslyn compilation server as well). This executable is the compilation server that the csc.exe executables send requests to. Having the long-running compilation VBCSCompiler.exe server model is essential for the largest performance benefits of Roslyn as it avoids the relatively costly continual respawning of the compiler itself. If the compilation server cannot be started, the csc.exe processes will fallback to do the heavy compilations themselves.

Largess

I will check using Roslyn and batch, In-place would not be helpful, because the CI build is running after git clean and the PR build is running after the shared bin directory is deleted (workspace clean = outputs in Azure DevOps pipeline). Rationale - not cleaning the shared bin allows for stale binaries which may poison the unit tests, in turn poisoning the PR build on that particular build agent. But maybe recognizing and deleting stale binaries is worth its pain, if incremental in place is so good. I will have to test it. – Ardellearden

@Ardellearden I was mistaken about batch (and really should have said, ensure that debug is disabled..). I ran some local tests against the the full code base (previously it was just one folder, that displayed much different results - perhaps lots of JScript?) and couldn't get past ~35m on a good run, so either I'm not correctly toggling release mode or it doesn't have nearly the performance improvement with the VBCSCompiler compile model (this is where Roslyn keeps the compiler process running so avoid compile executable startup cost). – Largess

The change to Roslyn however, is huge and even earned accolades by the deployment team as it's highly noticeable on an in-place compiled / warmed-up target site. (Here we use the IIS Compilation in CI/CD as a verification step, not as a binary deployment artifact.) I would expect Roslyn performance improvements to be 'ideal' on pure-CSHTML & C# WebForms, and less ideal on older JScript (that doesn't run through Roslyn). I'd be interested to see some performance numbers of switching to Roslyn on your code base. – Largess

I installed the NuGet package. It puts roslyn binaries under the bin\roslyn folder of each published Asp.Net application. But then what? How do I compile the Mvc views during the build? The documentation I have seen so far mention aspnet_compiler.exe. Does it mean aspnet_compiler.exe is going to use roslyn, because of the roslyn folder there? If so, then how do I verify it is being used? Because I do not see any difference in performance, so I must be missing something. – Ardellearden

@Ardellearden The web.config compilers section has to be updated as well. I've put some notes at the end of the answer. – Largess

I have a dedicated question here - #70388852, if that helps. – Ardellearden

Could you have a look at #71329684 , please? – Ardellearden

I am forced to demote this from the answer, because it does not really provide any actionable instructions on how to use Roslyn. – Ardellearden

@Ardellearden The largest single speed improvement is using Roslyn and the out-of-process compiler model, which I hope you’ve been able to utilize. The process and resource that I used for the switch are included in the post. – Largess

I am sorry, I did not understand the first two bullets in the instructions and hence could not proceed. I have uploaded a tiny web app here - github.com/MarkKharitonov/TinyWebApp. It is a fully working tiny example of how we run the asp.net compiler. I will try to process your instructions again using this tiny web app as the test bed. But if you could clarify the first two bullets that would be very helpful. It would be fantastic if you could show how to modify the aforementioned tiny web app to use Roslyn. Thank you very much. – Ardellearden

@Ardellearden Oh, I had not realized that using such was an issue! I can take a look at the skeleton in the next few days and show the required updates. – Largess

@Ardellearden I also added just the nuget package and was wondering whether it works (It had added appropriate web.config changes). All I saw was aspnet_compiler.exe and no instance of VBCSCompiler... But boy, it did help by bringing down precompilation process from 34 minutes to 14 minutes. However for other project I don't own and can't precompile I checked runtime compilation - it does have VBCSCompiler.exe and is running csc.exe from bin/roslyn. – Milburt

And my findings / performance measurements within dynamic and precompilation using roslyn compiler - dev.to/janisveinbergs/dynamics-crm-improve-58g9 – Milburt