Problem description
I am trying to debug a problem on a customer's server (Win 2012R2) where one of our .NET Web Applications is hosted in their IIS (version 8.5.9600.16384). Recently, one of our technical supports changed a minor internal setting in the Web.config file of the application and according to them, nothing else was changed or done to the system. The application was not updated either, and had been running without any issues before.
Since that change, the AppPool in which the site is configured cannot be started anymore. According to the System Event logs, there are a few attempted starts, each resulting in a worker process crash, until the Rapid-Fail Protection of the pool stops the startup process entirely.
Also, since I'm on a customer's server, I only have limited debugging options.
Findings
Reported in the System Event log by WAS:
A process serving application pool '[our pool]' suffered a fatal communication error with the Windows Process Activation Service. The process id was '5664'. The data field contains the error number.
Application pool '[our pool]' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
A look at the corresponding Application Error log reveals the following:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0 Faulting module name: PerfMon.dll, version: 8.0.10977.0, time stamp: 0x59cfb424 Exception code: 0xc0000409 Fault offset: 0x000f8c7b Faulting process id: 0x1620 Faulting application start time: 0x01d423ff75bef49f Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll Report Id: b56327b3-8ff2-11e8-80d5-005056a52299 Faulting package full name: Faulting package-relative application ID:
While googling for this problem I mainly focused on PerfMon.dll issues and the provided error code 0xc0000409, which is apparently either related to a corrupted registry entry or a Stack Buffer Overflow, but in both cases I've simply been unable to figure out what the cause could be.
I also tried the crash dump analysis described here https://blogs.msdn.microsoft.com/parvez/2016/08/06/iis-application-pool-crash-and-debug-diag/
But in the end it seemed like it was just a round-trip back to the original problem, as this was the only error found in the dump:
The assembly instruction at PerfMon!DllGetClassObject+966fb in C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll from Microsoft Corp. has caused an unknown exception (0xc0000409) on thread 2
Which is basically what I've gotten from the event log already.
My main question here is not only what might be the cause and a potential fix to this problem, but also how to properly find the cause/solution in the first place.
UPDATE
Using Fusion Logs, I was able to find three binding errors for System.Web.DynamicData, System.Web.Extensions and System.ServiceModel.web.
The operation failed. Bind result: hr = 0x80004005. Unspecified error
Assembly manager loaded from: >C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll Running under executable C:\Windows\SysWOW64\inetsrv\w3wp.exe --- A detailed error log follows.
=== Pre-bind state information === LOG: DisplayName = System.ServiceModel.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35 (Fully-specified) LOG: Appbase = [app root path] LOG: Initial PrivatePath = [app root bin directory] LOG: Dynamic Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a LOG: Cache Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a LOG: AppName = b83a4ee Calling assembly : (Unknown).
Rejecting code sharing because a dependent assembly did not match the conditional APTCA share mode
Searching for any information on this I was only able to find an old error related to MVC3/MVC4, which required a configuration change. That is unrelated to this issue, though. This might also mean that the original error has nothing to do with the PerfMon at all and that the original error is just a weird side effect.
C:\Program Files\Microsoft Monitoring Agent\Agent\...is not a required component in iis. Try searching for that path in your web.config or the application host config and comment it out? Guessing it's installed some module/handler and it's that that's failing? Could just need an update? – Salter