Crash dump - resolve unmanaged code crash in a .NET application using WinDbg

Question

I'm trying to discover the WinDbg tool to analyze a crash dump we have on our production server.

When I run !analyze -v, I get:

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/w3wp_exe/7_0_6002_18005/49e03238/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

FAULTING_IP: 
[phone]
00000000`00000000 ??              ???

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  00000000000029b0

DEFAULT_BUCKET_ID:  WRONG_SYMBOLS

PROCESS_NAME:  w3wp.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x29b0 (0)
Child-SP         RetAddr          Call Site

PRIMARY_PROBLEM_CLASS:  WRONG_SYMBOLS

BUGCHECK_STR:  APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER:  from 000000007749c0b0 to 00000000775e6d5a

STACK_TEXT:  
00000000`0012f6c8 00000000`7749c0b0 : 00000000`00000000 000007fe`faf07e6b 00000000`00000000 000007fe`f9c015f0 : ntdll!ZwWaitForSingleObject+0xa
00000000`0012f6d0 000007fe`f9c03e74 : 00000000`00000158 00000000`ffb35de0 00000000`00000000 00000000`00000158 : kernel32!WaitForSingleObjectEx+0x9c
00000000`0012f790 00000000`ffb3235a : 00000000`fffffffe 00000000`00000001 00000000`007e6400 00000000`0000008c : w3wphost!AppHostInitialize+0x280
00000000`0012f7f0 00000000`ffb33b71 : 00000000`00000000 00000000`ffb33ce5 00000000`00000000 00000000`00000000 : w3wp!wmain+0x466
00000000`0012f980 00000000`7748be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : w3wp!PerfStopProvider+0x199
00000000`0012f9c0 00000000`775c6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012f9f0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d


STACK_COMMAND:  ~0s; .ecxr ; kb

FOLLOWUP_IP: 
w3wphost!AppHostInitialize+280
000007fe`f9c03e74 f6052998000003  test    byte ptr [w3wphost!g_dwDebugFlags (000007fe`f9c0d6a4)],3

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  w3wphost!AppHostInitialize+280

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: w3wphost

IMAGE_NAME:  w3wphost.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  49e0420f

FAILURE_BUCKET_ID:  WRONG_SYMBOLS_80000003_w3wphost.dll!AppHostInitialize

BUCKET_ID:  X64_APPLICATION_FAULT_WRONG_SYMBOLS_w3wphost!AppHostInitialize+280

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/w3wp_exe/7_0_6002_18005/49e03238/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

Followup: MachineOwner

I really have a hard time figuring what is what. From what I understand, here are the interesting part:

EXCEPTION_CODE and STACK_TEXT.

I'm a really new to WinDbg, and it's the first time I'm using this tool. I've been struggling with my Google search, so I guess I'm not searching for the right thing.

What I'd like to do is:

1. Understand the output format of the stack_text
2. Try to see the input parameters of each functions

Is that the right way to approach this problem?

Prodrome

I have the same result of !analyze -v. Please tell me what was the problem? – Commentate

Answer 1

There are several good tutorials available on the web and even in the WinDbg help file (.chm). A good place would be WinDBG tutorial - Introduction or Tess' blog, If broken it is, fix it you should.

In your case, step 1 would be to point WinDbg to the correct symbols. It's clear from the output above that your sympath is either incorrect or not pointing to any PDB files. Do the following in the debugger:

.sympath SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

This will point the debugger to use the Microsoft public symbol server for OS components; it will cache the PDB files to your c:\symbols folder. To add another symbol path (for example, the folder containing your application's PDB files), you can either use a ';' delimited list of paths or simply use the .sympath+ command to add new paths piecemeal.

Once you set up your symbol path, run !analyze -v again or follow the steps in the tutorial above to see if you get better results.

Elisha

Thank you for your input. I'll start looking into that these tutorial right now. Although, I had already set up the symbols path in WinDbg, so I'm surprised that you say they are not loaded. How did you see it? And do you know what the problem might be? Path is set to a valid path on my computer and the URL is correct. – Prodrome

The ".symfix" command would set the symbol path to MS symbol server. Try "!sym noisy" to get symbol look up failures. – Lai

Sorry for the delay in responding. There were two telltale signs of wrong symbols. First, !analyze showed the obvious "WRONG_SYMBOLS" message as the primary problem. Second, the offset for the functions into w3wp are a little big (0x280 for one, 0x199 for the other). As Naveen said, !sym noisy followed by a .reload will allow you to get verbose info on where it's finding the pdbs and why it isn't. – Elisha

Answer 2

The stack trace should be readable if you have the correct symbols. You could try something like:

1. Load the dump file.
2. Run .symfix
3. Open the 'Symbol File Path' menu
4. Add a path to your application's .PDB files
5. Check the 'reload' checkbox
6. Run !clrstack -p to dump the stack with parameters.

Seminal