How is .NET JIT compilation performance (including dynamic methods) affected by image debug options of C# compiler?

Question

I am trying to optimize my application for for it to perform well right after it is started. At the moment, its distribution contains 304 binaries (including external dependencies) totaling 57 megabytes. It is a WPF application doing mostly database access, without any significant calculations.

I discovered that the Debug configuration offers way better (~5 times gain) times for most operations, as they are performed for the first time during the lifetime of the application's process. For example, opening a specific screen within the app takes 0.3 seconds for NGENed Debug, 0.5 seconds for JITted Debug, 1.5 seconds for NGENed Release and 2.5 seconds for JITted Release.

I understand that the gap in JIT compilation time is caused by the JIT compiler applying more aggressive optimizations for the Release binaries. From what I can tell, Debug and Release configurations differ by the /p:DebugType and /p:Optimize switches passed to the C# compiler, but I see the same performance gap even if I build the application with /p:Configuration=Release /p:DebugType=full /p:Optimize=false – that is, the same image debug options as in /p:Configuration=Debug.

I confirm that the options were applied by looking at the DebuggableAttribute applied to the resulting assembly. Observing the NGEN output, I see <debug> added to the names of some assemblies being compiled – how does NGEN distinguish between debug and non-debug assemblies? The operation being tested uses dynamic code generation – what level of optimization is applied to dynamic code?

Note: I am using the 32-bit framework due to external dependencies. Should I expect different results on x64?

Note: I also do not use conditional compilation. So the compiled source is the same for both configurations.

Collier

As your Release NGENed assemblies still are slower than Debug, are you sure that JIT is the problem ? You could try a profiler ... Also, check that you aren't using #if DEBUG in your code. – Saba

Are you using XmlSerializer without SGEN ? #772227 – Saba

I am not using #if DEBUG (edited question to reflect this). The application is not necessarily slower on Release - it may even be faster, but I am measuring cold startup time, not throughput. I suspect the JITting of dynamic methods and so I ask what decides the optimization level of those. – Collier

This all makes little sense. These are warm start numbers, ngen makes cold starts slower. I guess you ought to try using the [Debuggable] attribute explicitly. – Harney

@HansPassant - what do you mean? The times I've given are measured as follows: after the application loads and welcome screen ("launchpad") appears, a tile is clicked, and time is counted until the requested screen appears. That's "cold" startup for me. "Warm" is when I click "Back" (the app is navigation-based) and click the same tile again (no "caching" of screens is being done by my code). The "warm" performance is satisfactory in all configurations. – Collier

No, cold start is when you run the app for the very first time after a boot. Dominated by finding the files on disk. Ngen makes it worse because it doubles the number of files. Which is why Microsoft recommends to not ngen small assemblies. You are measuring warm start here. Yes, ngen speeds that up. No jitting and no optimization. – Harney

@Saba - I SGEN'ed and NGEN'ed the sole assembly in the solution that contains types used with XmlSerializer and observed no changes in performance. – Collier

NGEN images are not used always used (when loading assemblies with Assembly.LoadFrom for instance) it might be the case for some of your assemblies. – Saba

@Saba - unfortunately not. All assemblies are implicitly loaded in the Load context from the entry assembly. – Collier

Try opening your project file in notepad or a similar text editor and seeing if there's another difference in the compile settings that you're overlooking (usually it's one that is hidden from the screens). – Bobbysoxer

Answer 1

If, as you say, you have 304 assemblies to be loaded, then this is likely a cause of your app running slow. This seems like an extremely high number of assemblies to be loading.

Each time the CLR reaches code from another assembly that's not already loaded in the AppDomain, it has to load it from disk.

You might consider using ILMerge to merge some of those assemblies. This will reduce the delay in loading the assemblies from disk (you take one, larger, disk hit up-front).

It may require some experimentation, as not everything likes being merged (particularly those which use Reflection, and depend upon the assembly filename never changing). It may also result in very large assemblies.

Hamford

I'll try, but I think that's unlikely the cause of the speed difference between JITted Debug and NGENed Release configurations. I run the tests on a SSD machine. – Collier

Answer 2

Okay, a few questions here.

From what I can tell, Debug and Release configurations differ by the /p:DebugType and /p:Optimize switches passed to the C# compiler, but I see the same performance gap even if I build the application with /p:Configuration=Release /p:DebugType=full /p:Optimize=false – that is, the same image debug options as in /p:Configuration=Debug.

Although the tick boxes are the same, changing to Release mode also causes certain internal code paths to be removed, like Debug.Assert() (used heavily in the Microsoft internal code). So these are not evaluated at runtime, which causes some performance improvement. DebugType=full generates a PDB matching the code it was compiled against, so isn't a performance hit in itself. If the PDB is deployed, the exception handling code will use the PDB to provide more useful stack traces against the compiled code. Release mode also internally triggers some memory improvements, because Debug versions are used to attach the debugger.

NGEN is a tool is used to 'potentially' optimise an application. It optimises the code to work on specific to the computer you are on. But it can have drawbacks, as the JIT compiler can make changes to the layout of the code in memory, whereas NGEN is more static in its nature.

As for 32-bit (x86) dependencies, your app will now run in x86 mode. If the dependency had both x86 and x64 version available, and if your app is compiled into under an 'Any CPU' compilation mode, the JIT compiler will switch automatically between the 2. NGEN would only generate a specific version for the current computer. So if you did NGEN and then distribute, it would only work for the specific architecture you compiled against.

If you are not utilising conditional compilation features, it does not really matter if you switch from Debug to Release. But you will see a performance benefit in Release.

With the NGEN, I suggest you test extensively to see the benefits over the 2. It doesn't always result in better performance.

Situation

I do believe that calls to ConditionalAttributed methods are stripped at the stage of compilation to IL, not JIT. Regardless, the difference in my case is not between JIT/NGEN but Debug/Release. – Collier

So are you asking what the difference is between Debug and Release if the configuration is the same? – Situation

Yes. Are there extra flags in PE (apart from DebuggableAttribute)? And most importantly, what level of optimization is applied to dynamically generated code? – Collier

I don't think there is? Release builds are just a pre-configured set of options to make your application run optimally. ASP.NET applications work differently as they are generated on the fly. – Situation

Answer 3

Are you running it under the debugger ('F5') or without the debugger ('ctrl+F5')? If the former, ensure Tools -> Options -> Debugging -> "Suppress JIT optimization on module load" is unchecked

Terrieterrier

I do the measurements running the executable from a command line. So that's not it. – Collier

Hrm. I guess that also wouldn't make it slower than DEBUG. – Terrieterrier