How to find the source of an 'Access Violation'

Question

To put in a nutshell, I have a C# application doing lots of mciSendString calls ( via dllimport ) to control wav files playback ( essentially open, play, pause, stop, status, close ). And after running for a while, the application crashes without notice with an 'access violation'.

Even though I'm running the app from my vs2012, the exception is not caught by visual studio. Even with the 'force break on an exception' option, I've had no luck in debugging this from the vs2012. So instead I've setup WER to generate me crash dumps and I am using windbg with psscor2.dll plugin to debug it.

Then in sequence, using the following commands, this is what I get ( shorten to the essential for readability purposes ) :

$>.ecxr

eax=00000001 ebx=00000000 ecx=00000401 edx=00000000 esi=049725b8 edi=00000002
eip=4e88159e esp=0a4efa38 ebp=0a4efa54 iopl=0         nv up ei pl nz ac pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216
<Unloaded_mciwave.dll>+0x159e:
4e88159e ??              ???

$>~*kb

#  19  Id: 105c.28cc Suspend: 1 Teb: 7ef06000 

Unfrozen
user32!NtUserGetMessage+0x15
user32!GetMessageA+0xa1
winmm!mciwindow+0x102
kernel32!BaseThreadInitThunk+0xe
ntdll!__RtlUserThreadStart+0x70
ntdll!_RtlUserThreadStart+0x1b

# 30  Id: 105c.15f8 Suspend: 0 Teb: 7ef1b000 Unfrozen
ntdll!ZwWaitForMultipleObjects+0x15
KERNELBASE!WaitForMultipleObjectsEx+0x100
kernel32!WaitForMultipleObjectsExImplementation+0xe0
kernel32!WaitForMultipleObjects+0x18
kernel32!WerpReportFaultInternal+0x186
kernel32!WerpReportFault+0x70
kernel32!BasepReportFault+0x20
kernel32!UnhandledExceptionFilter+0x1af
ntdll!__RtlUserThreadStart+0x62
ntdll!_EH4_CallFilterFunc+0x12
ntdll!_except_handler4+0x8e
ntdll!ExecuteHandler2+0x26
ntdll!ExecuteHandler+0x24
ntdll!RtlDispatchException+0x127
ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
<Unloaded_mciwave.dll>+0x159e

#  31  Id: 105c.2310 Suspend: 1 Teb: 7ef00000 Unfrozen
user32!NtUserGetMessage+0x15
user32!GetMessageW+0x33
mciwave!TaskBlock+0x1d
mciwave!PlayFile+0xcb
mciwave!mwTask+0x98
winmm!mmStartTask+0x22
kernel32!BaseThreadInitThunk+0xe
ntdll!__RtlUserThreadStart+0x70
ntdll!_RtlUserThreadStart+0x1b:

$>!analyze -v

FAULTING_IP: 
mciwave_4e880000!TaskBlock+1d
4e88159e ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 4e88159e (mciwave_4e880000!TaskBlock+0x0000001d)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000008
   Parameter[1]: 4e88159e
Attempt to execute non-executable address 4e88159e

PROCESS_NAME:  Titan.vshost.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1:  00000008

EXCEPTION_PARAMETER2:  4e88159e

WRITE_ADDRESS:  4e88159e 

FOLLOWUP_IP: 
mciwave_4e880000!TaskBlock+1d
4e88159e ??              ???

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x15f8 (30)
 ====> Exception cxr@a4ef750

FAULTING_THREAD:  000015f8

BUGCHECK_STR:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_WRONG_SYMBOLS

PRIMARY_PROBLEM_CLASS:  SOFTWARE_NX_FAULT_CODE

DEFAULT_BUCKET_ID:  SOFTWARE_NX_FAULT_CODE

LAST_CONTROL_TRANSFER:  from 4e881999 to 4e88159e

STACK_TEXT:  
0a4efa54 4e881999 0a4efa88 078db198 078db1a4 mciwave_4e880000!TaskBlock+0x1d
0a4efa68 74370ae5 00038edc 00000000 00000000 mciwave_4e880000!mwTask+0x45
0a4efa88 7670338a 078db198 0a4efad4 76f99f72 winmm!mmStartTask+0x22
0a4efa94 76f99f72 078db198 79f84a28 00000000 kernel32!BaseThreadInitThunk+0xe
0a4efad4 76f99f45 74370ac3 078db198 00000000 ntdll!__RtlUserThreadStart+0x70
0a4efaec 00000000 74370ac3 078db198 00000000 ntdll!_RtlUserThreadStart+0x1b


SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  mciwave!TaskBlock+1d

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: mciwave_4e880000

IMAGE_NAME:  mciwave.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bcb4a

STACK_COMMAND:  ~30s; .ecxr ; kb

FAILURE_BUCKET_ID:  SOFTWARE_NX_FAULT_CODE_c0000005_mciwave.dll!TaskBlock

BUCKET_ID:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_WRONG_SYMBOLS_mciwave!TaskBlock+1d

Followup: MachineOwner
---------

Exception seems to happen in thread #30 in Unloaded_mciwave.dll, but I have no idea how to push further the debugging.. How can I get a better idea of what's going ??

How can I get what's happening between these two lines ?

ntdll!KiUserExceptionDispatcher+0xf
--> WARNING: Frame IP not in any known module. Following frames may be wrong.
<Unloaded_mciwave.dll>+0x159e

Thank you for your help in advance.

Faint

What does !analyze -v tell you? – Goldshell

@LievenKeersmaekers, I've edited my question with the command dump. – Faint

I find it curious that the analyze fase mentions thread 29 while there's no trace of that thread in your ~*kb result. Also the Unloaded_mciwave.dll strikes me as odd. To me, it looks like you are making a call into the dll when the dll is no longer in memory. <disclaimer>I'm a windbg novice myself so take with a (grain)bag of salt</disclaimer> – Goldshell

Oh I'm sorry, you're right I took the !analyze -v dump from a different .dmp file. I've re-updated my question with the proper command dump. Sorry again. Regarding this Unloaded_mciwave.dll , could it simply be referring a function unloaded in the mciwave dll ( as shown by the line SYMBOL_NAME: mciwave!unloaded+159e ? ps: I'm also a novice with windbg – Faint

<Unloaded_mciwave.dll>+0x159e definitely means that the program is trying to call stuff from an unloaded DLL. Most likely you have callback registered somewhere, where the callback body came from the DLL. – Relive

Strange.. the only registered callback I can see are being registered with the mciwave API calls waveInOpen() and waveOutOpen(). I don't understand how we can reach this situation since the API is in full control of this. How can the dll get unloaded if it's in the middle of a callback.. Is there any way to get more info on what lead to this situation with windbg ? – Faint

Answer 1

You should get more details by reloading the DLL in the Debugger.

For that you need to do:

lmvm mciwave.dll
start             end                 module name

Unloaded modules:
e6510000 e6548000   mciwave.dll
    Timestamp: Fri Oct 14 12:00:00 2011 (4E98E6E2)
    Checksum:  0003E937
    ImageSize:  00038000

You need to set up the Symbol and Exe-Path so the debugger can find the DLL and PDB (which shouldn't be a problem if you have it in your machine). Then you can do

.reload mciwave.dll=e6510000,00038000
DBGHELP: <path>\mciwave.dll - OK

Now if you do !analyze -v again, it should give you the correct call stack.

Relive

+1. On my system this seems to lead to the following call call dword ptr [mciwave!_imp__GetMessageW (4f1810a0)] in mciwave!TaskBlock+0x5. Ofcourse, my original disclaimer from above applies :) – Goldshell

this is definitely worth a read – Goldshell

@tehlexx, it doesn't seem to give me more details. But I wonder if I got your instructions right. When I do the reload, should I specify the full path there as follow .reload C:\windows\system32\mciwave.dll=<start-address>,<image-size> ? And what is this DBGHELP: <path>\mciwave.dll is it the output of the previous command ? If so, it gives me nothing :S – Faint

@tehlexx, Actually when I do lmvm mciwave.dll I do see a lot of mciwave.dll loaded. I must be doing something really wrong in the code to have so many instances of it, or is it something normal ? Also, I understand now what you meant by the symbol-and-exe-path ( that's the one you either specify by command-line -y option, or .sympath command right ? ). Well in any case, the .reload is not verbose. I actually don't know if it worked. @LievenKeersmaekers, I'm a bit lost. How did you reach this conclusion ? Also it seems that you're spot on. Look at the thread #31's callstack. – Faint

The start-address and image-size arguments should be taken from the unloaded module output. You can set symbol loading to verbose with !sym noisy. I've seen cases of multiple instances of a DLL loaded, not sure if that automatically is a bug. – Relive

Yes I did get start-address and image-size arguments from the unloaded module output, and also had !sym noisy enabled too. – Faint

Strange. You could also try adding /f (force) – Relive

@tehlexx, you are a genious ! /f made the difference in my case. I'm gonna update my question with the new !analyze -v dump. – Faint

May I pick further your brain and ask you how to find out what the app was attempting to do with this mciwave_4e880000!TaskBlock+0x1d ? I now have a better callstack but I still have no clue of what to do with that :( Trying to do a dds on this address and it doesn't give me any information.. Do you any suggestions ? – Faint

No specific ideas, sorry. An educated guess would be that winmm!mmStartTask+0x22 still had a task scheduled when mciwave.dll was already unloaded, I'd start investigation with a) are you scheduling tasks? and b) why was the DLL unloaded? – Relive