Why is ASP compiling my views so frequently?

Question

We have 4 servers load balanced:

• 4 cores @ 2.6Ghz (E5-2650 v2)
• 14GB RAM
• Windows 2012 R2
• High Performance power setting
• IIS 8.5
• ASP 5.3
• EF 6.1

They each have a single application pool with one worker process and a single website. Each server has its own copy of the site (DLLs & views), running on a local disk. We are using IIS virtual directories to point to shares on a clustered file server for log files and common images etc (content only). The application pools are set to not shut down when idle (interval of 0) and we have also disabled the every-1740 minute recycle interval too.

We have New Relic's .NET agent installed on all servers, and looking through our slow transaction log, I can see that many requests are taking 15 seconds or so to complete. Looking into the trace, I can see a common call to System.Web.Compilation.AssemblyBuilder.Compile() and System.Web.Compilation.BuildManager.CompileWebFile().

As far as I know or understand, ASP would compile these views upon first request to them, and cache it (to the temporary ASP files in C:\Windows\Microsoft.Net) and then load from there for subsequent requests.

I'm confused how this is happening so often - when I visit these URLs, the TTFB is about 400ms, and due to constant load I can't see the websites "losing" their cache and needing to compile the views again. These pages are frequently hit - it's an e-commerce store and I can see that it happens often, and on our most popular pages: catalogue (category/brand/gender etc) listings and product details.

I've set the settings against each application pool to log an event when recycling, and there have been no events logged when I'm checking the WAS service in the event viewer. We also have New Relic server installed, and looking over the past 6 hours' worth of data, I can't see any dip in RAM usage on any of the servers - which would indicate the application pool recycling. This has really baffled me!

I'm thinking of moving towards pre-compiling our views as part of our release process - it makes sense really. But it feels like that is working around, or masking an issue which as far as I can see should not be happening. We build our site in Release mode and have <compilation debug="false" /> on all web.config files.

Can anyone think of any causes for this?

Allotrope

How did you come to a conclusion that System.Web.Compilation.AssemblyBuilder.Compile() and System.Web.Compilation.BuildManager.CompileWebFile() are consuming most of the 15 secs delay? Just wondering if the delay is somewhere else. Have you tried to capture how much time is spent after an action method receives a request? What about the network latency between the client making http request and the server? – Triglyceride

I'm using New Relic to see 'slow transactions' and all of our slow transactions contain calls to these methods and it takes 15% of the page request. I see a similar effect when I'm developing on the site - when I rebuild my project and reload a page, it takes a lot longer to render (putting aside application startup times). If I then visit a page on a different controller then there's a noticeable "slowdown" vs refreshing that page again (while the view is rendered for the first time) – Allotrope

Not trying to deviate from the actual topic, but most of the applications I have been working on recently using asp.net mvc, I am pushing them more towards single page application architecture. May not be pure SPAs, I still have multiple views, but lot less of them. This design helps me avoid the kind of issue you are facing. It introduces complication on the client side, but much better user experience. – Triglyceride

Answer 1

It is because of how JIT (Just-In-Time) compilation works.

When you build your application, it is converted into .NET Microsoft Intermediate Language (MSIL) or Intermediate Language (IL).

As your applications is accessed, Common Language Runtime (CLR) converts only executed IL parts of your code into native instructions.

Just-In-Time compilation process converts IL to native machine instructions and it is a part of CLR.

In a simplified terms when you run a .NET application and your program calls a method. JIT Compiler reads IL from metadata and compiles it into native instructions and run it. Next when your program calls the same method, CLR executes native CPU instructions directly. This process adds some overhead for the first method call. You can go with the other option of pre-compiling your application using NGEN, which is usually not recommended because you will loose some optimizations that only JIT can perform due to its awareness of underlying hardware platform. These two articles has more details http://geekswithblogs.net/ilich/archive/2013/07/09/.net-compilation-part-1.-just-in-time-compiler.aspx and https://msdn.microsoft.com/en-us/library/ms366723.aspx

There are also other things you can try that might help you speed up your application. You can use IIS Application warm up module How to warm up an ASP.NET MVC application on IIS 7.5?, implement distributed caching etc to alleviate some of your application bottlenecks.

Triglyceride

Hi, thanks for the answer - but I do see System.Web.Compilation.BuildManager.CompileWebFile() being called often - on MVC controllers which would receive at least 1/request a second. Am I incorrect in thinking that all the "hot spots" would have been frequently hit during the application's running time (currently it's about 5 hours since the application pools were last recycled)? – Allotrope