ASP.NET keeps compiling

Question

After deploying our ASP.NET (mostly forms with AJAX) applications to one particular production machine, the recompiling process (where w3wp.exe calls CSC.exe for each aspx or ascx file) continues for hours. The performance of the web app is therefore very sluggish. These very same applications when deployed to other production machines finishes its compiles in minutes.

These aspx compiles are only visible in ProcExp for a second, and the commandlines they use involve a parameters file that is deleted immediately after compiling, so it is very hard for me to monitor what files are being compiled, or to see if the same file is being compiled more than once. Is there any way to log the compilation process so I can try to figure out why this one machine misbehaves?

I know that we can use aspnet_compiler.exe to precompile the entire application, but getting that into the build and deploy workflow (TFS 2010 plus OctopusDeploy) will be a bit of a pain. Furthermore, my attempts to do the precompilation "manually" at the command line aren't working. The precompilation suceeds, but when IIS is pointed at the target directory, any request for any file is throwing this sort of error:

Exception type: HttpException

Exception message: The file '/default.aspx' has not been pre-compiled, and cannot be requested.

1. Is there something I should check to determine why this machine spends so much time compiling?
2. Is there something I can do to log which files are being compiled when?

Agger

You have given a wide range of things you've tried... so it's hard to answer the question because I don't know what you actually want... – Squaw

I finally found a tool called FrontMan that lets me at least capture the arguments AND the contents of the parameters file. I finally can see that most of the time is spent working on resx files. If this works out I'll promote this to an answer – Agger

Sorry folks. No bounty will be awarded, because none of your answers were even as helpful as the FrontMan utility ... which itself is not an answer because using it causes some of the compilations to fail, leading to the app to fail. – Agger

Answer 1

We use precompilation as the only method to solve this issue, otherwise you'll see multiple msbuild.exe processes. On the fly compilation cannot give you the desired result. With Visual Studio 2012 or later it's really easy to create a publish profile and then commit that profile to source control so that you don't have to do it again and again. Also you blame the build process for the bad performance of your application, I would suggest taking a look into how to optimize your app itself. Like for us we were also having a sluggish performance, then we switched to SessionPageStatePersister which improved a lot of performance issues. I would recommend the following awesome videos, just go through them and check how many you've already done and how many you can do.

Performance Optimize Your ASP.NET Web App

Deep Dive: Improving Performance in Your ASP.NET App

Master

There are ZERO msbuild processes at ANY time. Precompilation probably is the only solution I should actually pursue -- "nuke the site from orbit, it's the only way to be sure" -- but I can't award you the bounty based on this answer. – Agger

It was not for the bounty. Hope the videos provided some help. – Master

Answer 2

Dynamic Compilation

ASP.NET dynamic compilation should keep files in a cache after compilation and recompile only when something changes. As MSDN says:

Any changes to a dynamically compiled file will automatically invalidate the file's cached compiled assembly and trigger recompilation of all affected resources

Most likely in your case, the dynamic compilation affects itself in a way that triggers recompilation like - in a loop.

Did you check the optimization of dynamic compilation ?

If you want to be able to change top-level files without causing the whole site to be recompiled, you can set the optimizeCompilations attribute of the compilation element in the Web.config file to true.

Note: For those in .NET Framework 3.5, they'll need a hotfix to use that parameter.

Logging

Do you have already a logging solution plugged into asp.net framework ?

The minimum to be able to investigate issues is either a ConsoleLogger (visible from web browser) or a Logger writing log events to a file.
You must then configure LogLevel to the minimum: Debug or Verbose should do the trick.

There are also several third parties logging framework that could be plugged into System.Diagnostics to provide more features.

General advices

If you have a bug issue, like recompile in loop, you should try to reproduce it on a developer machine. It will be easier to debug step-by-step the recompilation process. This article on what occurs during recompilation can help you. On the whole deployment and compilation process, this is even more complete but it's for ASP.NET 2.0

If you have performance issue, you should profile your website on a developer machine, pin-point where the real performance issues are and fix them.

Columbuscolumbyne

If the issue was dynamic compilation affecting itself in a way that triggers recompilation -- then it would be expected to manifest on ALL production machines, not just one. – Agger

optimizeCompilation attribute can't be the answer. There are no changes happening to ANY files -- every file in the app directory is read-only and no timestamps or file sizes are changing. It NEVER goes back into this recompile loop unless a file actually changes. When the app "Settles down" it stays settled until the next deployment -- not the next app pool recycle or even machine reboot. – Agger

The performance issues cannot be solved on a dev machine. The performance issue is specific to this production machine, and clearly connected to csc.exe processes. – Agger

Answer 3

• Clean the temporary directory in Microsoft.Net framework
• Run chkdsk on disk
• Temporary disabile antivirus/antispyware
• Check with sysinternal filemon tool if someone (other csc.exe and w3wp.exe) access to temporary dotnet directory

Center