How can I determine the cause of an IIS Crash on a 64-bit Server

Question

I have a .net 2.0 web application that is running on Windows server 2003 Standard x64, using IIS 6.

The application pool for our website started crashing recently and I can't determine why. It started happening on a weekend, and the latest release of the website was several days earlier. I have determined that no other changes were made to the server recently, including code and Microsoft updates.

The event log shows the following whenever a crash occurs with no additional information in the data block:

Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d691cc, faulting module kernel32.dll, version 5.2.3790.4062, stamp 462643a7, debug? 0, fault address 0x0000000000027d8d.

This is running on a x64 server so I can't use any of the standard Debug Diagnostic Tools because even though there is a 64 bit version of it, it only attaches to IIS running in 32 bit mode.

I have tried using the Debugging Tools for Windows (x64) and was able to attach to the w3wp process, and waited for another crash. However, this slowed the server down so much that it was unusable, so I had to stop it.

What other methods can I use to try to determine the cause of an IIS crash?

Sewel

Answer 1

Read about ASP.NET 2.0 Crash case study: Unhandled exceptions.

Strategy #1 – logging the exception
The first way, and this is the way I would probably recommend, is to create an UnhandledExceptionHandler to log the exception along with its stack trace in the event log as shown in this article http://support.microsoft.com/?id=911816 You add the handler like this to the web.config:

<system.web>
  <httpModules>
    <add type="WebMonitor.UnhandledExceptionModule, <strong name>"
       name="UnhandledExceptionModule"/>
  </httpModules>
      …
</system.web>   

And it hooks an eventhandler up to the UnhandledException event of the current app domain. You don’t actually need to strong name it and add it to the GAC, however if you plan it in multiple applications you should to avoid for the dll being loaded multiple times. Now the next time you get one of these unhandled exceptions, the process will still exit (unless you change the unhandled exception policy), but you have a very good chance of fixing the issue.

Bridgeman

Good find, I'm trying this now. – Sewel

This article didn't directly work for me, but is the correct answer. Because this affected a production system, I opened a support ticket with Microsoft. They first pointed me to this same article, and then supplied me with a debug tool to help find the actual problem. The problem was an endless recursive loop, because of poorly nested data. – Sewel

Answer 2

Microsoft's Debug Diagnostic Tool (DebugDiag) will do the trick. It will provide an IIS memory dump and analysis.

Clie

Answer 3

You can set up performance counters to monitor things like CPU, memory and .NET specific counters. There are a lot of details, but this TechNet article might help:

• Monitoring ASP.NET Performance (IIS 6.0)

ASP.NET itself has a whole namespace for monitoring application health. You can create your own events or, most commonly, configure your application for the default events. This MSDN article has more:

• ASP.NET Health Monitoring Overview

If the problem is the application code such as an unhandled exception (although if this were your problem I'd expect to see more details in the Windows Event Log), you can use tools to trap and report on them. ELMAH is a great tool that I've used in the past for this. It's been described as Tivo for web applications and has a variety of ways to deliver details of exceptions and help track down what's wrong.

Siphonostele