Windbg crash dump analysis

Question

I'm having a hard time getting any meaningful information from a crash dump I created with ProcDump, but I'm pretty sure it's relevant to a seemingly random crash I've been having.

I have a VB6 application running on Windows 7 64-bit. Every once in a while, it crashes, leaving an entry in the error log that faults ntdll.dll but gives no more information than that. So, I have been running the process with SysInternals' ProcDump running to automatically create crash dumps for me.

I've been unable to re-create the crash in-house, so I was pretty sure that if I had a dump at all, it would tell me what the issue was. However, after running for most of a day, I see that ProcDump wrote several dumps already, although the program is still running fine. It does seem to point to issues with ntdll.dll, but I have no idea where to begin applying a fix for this.

Running !analyze -v on one of the dumps gives me the following:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


FAULTING_IP: 
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  000007c8

PROCESS_NAME:  application.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  application.exe

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT_AFTER_CALL

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT_AFTER_CALL

LAST_CONTROL_TRANSFER:  from 7754431f to 7752014d

STACK_TEXT:  
0382fdf4 7754431f 00000005 035e62c8 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0382ff88 74cd339a 00000000 0382ffd4 77539ed2 ntdll!TppWaiterpThread+0x33d
0382ff94 77539ed2 035e6298 74e2a30c 00000000 kernel32!BaseThreadInitThunk+0xe
0382ffd4 77539ea5 775441f3 035e6298 00000000 ntdll!__RtlUserThreadStart+0x70
0382ffec 00000000 775441f3 035e6298 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~0s; .ecxr ; kb

FOLLOWUP_IP: 
ntdll!ZwWaitForMultipleObjects+15
7752014d 83c404          add     esp,4

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!ZwWaitForMultipleObjects+15

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce7ba58

FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_AFTER_CALL_80000003_ntdll.dll!ZwWaitForMultipleObjects

BUCKET_ID:  APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_ntdll!ZwWaitForMultipleObjects+15

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/BlackJack_exe/1_5_0_0/50227d4e/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

Followup: MachineOwner

Can anyone point me in the right direction, in terms of making sense of this entry, and what I can do about it?

Plica

Where you debugging this application while procdump was running? The APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL would seem to indicate that a debugger broke into the process and that is the dump that procdump captured. So these dumps aren't going to help you as they aren't snapshots of the actual problem. ProcDump is a very light-weight tool you should try running it where ever the actual problem is occurring and then try to analyze those dumps. – Dollydolman

I was not debugging at the time. I just installed ProcDump in-place on a production machine, scripting it to run (C:\apps\procdump.exe -accepteula -e -h -n 10 -t -w application.exe C:\application.dmp) at startup. So basically, I'm running ProcDump alongside our compiled executable, catching dumps as they come along. So what you're saying is these dumps are basically being caused by ProcDump, so I don't need worry about them? The error that I'm trying to catch most definitely ends in program termination, so if it happens in the field, that's the dump I'm trying to get anyway. – Plica

I'm not saying you don't need to worry about them. Just that the report of !analyze -v made me think these weren't actual crashes but instead debugger breaks. I would inspect the call stacks of the various threads and see if there is another exception somewhere. – Dollydolman

I didn't see any other exceptions by looking at the call stacks, so I'm hoping that this is good. – Plica

Answer 1

Just to make sure, I've performed some tests on my side, attaching to healthy processes and making dumps of just started process. In all cases output of !analyze -v was pretty similar to yours, except of the fact that my one is more verbose, I think it depends on debugger version.

For example, here is the output that I've got after attaching to just started Paint:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

GetPageUrlData failed, server returned HTTP status 404
URL requested:     http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1

FAULTING_IP: 
ntdll!DbgBreakPoint+0
00000000`76d90530 cc              int     3

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000076d90530 (ntdll!DbgBreakPoint)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 0000000000000000

FAULTING_THREAD:  0000000000000cbc

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT

PROCESS_NAME:  mspaint.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

EXCEPTION_PARAMETER1:  0000000000000000

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

LAST_CONTROL_TRANSFER:  from 0000000076e37ef8 to 0000000076d90530

STACK_TEXT:  


FOLLOWUP_IP: 
ntdll!DbgBreakPoint+0
00000000`76d90530 cc              int     3

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!DbgBreakPoint+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ec4aa8e

STACK_COMMAND:  ~8s ; kb

FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_80000003_ntdll.dll!DbgBreakPoint

BUCKET_ID:  X64_APPLICATION_FAULT_STATUS_BREAKPOINT_ntdll!DbgBreakPoint+0

WATSON_STAGEONE_URL:      http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1

Followup: MachineOwner
---------

I've also taken a look at the explanation of ProcDump flags here: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx. It looks like using command line like

C:\apps\procdump.exe -accepteula -e -h -n 10 -t -w application.exe

you make procdump stop on every sign of hanging or exception without setting specific parameters, like memory usage number or procent of CPU usage.

I would suggest to use DebugDiag, which provides nice UI, where you could configure rules describing when dump should be created. Here is an explanation from me, how to gather a dump when you have excessive memory usage problem, or high CPU usage:

http://kate-butenko.blogspot.com/2012/06/how-to-gather-dump-with-debugdiag.html

and here is another fine screenshot-based explanation, how to get a dump in DebugDiag for specific exception:

http://blogs.msdn.com/b/kaushal/archive/2012/05/09/using-debugdiag-to-capture-a-dump-on-first-chance-exception.aspx

From set of more lightweight tools you may also check the AdPlus tool (resides in C:\Program Files\Debugging Tools for Windows (x64) folder). I prefer DebugDiag, as it allows to catch specific types of exceptions.

Amidships

Thank you for the thorough answer! – Plica