Determine line number of InnerException from minidump using WinDbg

Question

I'm trying to track down a NullReferenceException from a dump. The NullReferenceException is not the crashing exception, rather the crashing exception is a TargetInvocationException with an InnerException which is the NullReferenceException.

I'm using Windbg with SOS, I use the the command analyze -v and this gives me the call stack of the NullReferenceException:

EXCEPTION_OBJECT: !pe f6cb150
Exception object: 000000000f6cb150
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
SP               IP               Function
000000002CD9D8C0 000007FF01E7C639 MyDll!DoSomething2()+0xe99
000000002CD9DBE0 000007FF01E7B11D MyDll!DoSomething1()+0x43d
000000002CD9DD20 000007FF01E7AB11 MyDll!WorkerDoWork(System.Object, System.ComponentModel.DoWorkEventArgs)+0x51
000000002CD9DD80 000007FEEA68A0F2 System_ni!System.ComponentModel.BackgroundWorker.WorkerThreadStart(System.Object)+0x62

Notice, that I get the method names with byte offsets, but no line numbers. DoSomething2 is a large function, so it's not obvious where the NullReferenceException occurred.

I attempted to follow the instructions in Tess Ferrandez's blog:

.Net exceptions - Tracking down where in the code the exceptions occurred

But I got stuck early on where I attempt to determine the method descriptor for the method DoSomething2 using !ip2md with the IP of DoSomething2: 7FF01E7C639:

> !ip2md 7FF01E7C639
Failed to request MethodData, not in JIT code range

Note that the !ip2md command succeeds on the IP of the method where the TargetInvocationException occurred.

Question:

Where can I go from here to narrow down what line in DoSomething2 is crashing? Note that I cannot reproduce the crash, so all I have this (and several duplicate) dumps.

Additional Notes:

• .NET 4.0
• Windbg Version: 6.12.0002.633 AMD64
• I'm new to Windbg: so the more info, the better

Edit 1

When I don't have symbols set up properly, I get the following:

STACK_TEXT:  
00000000`2cd9d8c0 00000000`ffffffff MyDll!Unknown_0xe99+0xe99
00000000`2cd9dbe0 00000000`ffffffff MyDll!Unknown_0x43d+0x43d
00000000`2cd9dd20 00000000`ffffffff MyDll!Unknown_0x51+0x51
00000000`2cd9dd80 00000000`ffffffff system_ni!    System.ComponentModel.BackgroundWorker.WorkerThreadStart+0x62

When I set it up to point to my symbol server and turn on !sym noisy, it appears to load symbols properly:

0:000> ld MyDll
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\MyDll.dll - file not found
SYMSRV:  c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/MyDll.dll/4F3D6F4B154000/MyDll.dll not found
SYMSRV:  \\mysymbolserver\store\Mydll.dll\4F3D6F4B154000\file.ptr
SYMSRV:  MyDll.dl_ from \\mysymbolserver\store: uncompressed
DBGHELP: c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll - OK
DBGENG:  c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll - Mapped image memory
SYMSRV:  c:\symbols\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\Mydll.pdb not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/Mydll.pdb/8AFC2BE7529A41289FA9FBCEDB6836161/MyDll.pdb not found
SYMSRV:  \\mysymbolserver\store\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\file.ptr
SYMSRV:  MyDll.pd_ from \\mysymbolserver\store: uncompressed
DBGHELP: MyDll - private symbols & lines 
     c:\symbols\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\MyDll.pdb
Symbols loaded for MyDll

Edit 2

I tried using !name2ee as follows:

0:000> !name2ee MyDll!MyType.DoSomething2
Module:      000007ff004995b8
Assembly:    Autodesk.DataManagement.Client.Framework.Vault.dll
<invalid module token>

So, no luck there. But then I almost seemed to get somewhere with this:

0:000> !name2ee MyDll.dll!MyNamespace.MyType
Module:      000007ff004995b8
Assembly:    MyDll.dll
Token:       000000000200008c
MethodTable: 000007ff01b2e258
EEClass:     000007ff01b415e0
Name:        MyNamespace.MyType

0:000> !dumpmt -md 7ff01b2e258
EEClass:      000007ff01b415e0
Module:       000007ff004995b8
Name:         MyNamspace.MyType
mdToken:      000000000200008c
File:         C:\Program Files\MyCompany\MyProduct\Bin\MyDll.dll
BaseSize:        0x30
ComponentSize:   0x0
Slots in VTable: 31
Number of IFaces in IFaceMap: 2
--------------------------------------
MethodDesc Table
           Entry       MethodDesc      JIT Name
000007feeb31a2c0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007feeb3689f0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007feeb3688c0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007feeb353440 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01b01300 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01e89140 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01b9c080 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01f45f40 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01a9b358 000007ff01b2e128     NONE MyType.DoSomething3()
000007ff01a9b360 000007ff01b2e130     NONE MyType.DoSomething4()
000007ff01a9b368 000007ff01b2e138     NONE MyType.DoSomething5()
000007ff01e79800 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff020fea80 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01a9b3b0 000007ff01b2e1b0     NONE MyType.DoSomething6()
000007ff01a9b3b8 000007ff01b2e1b8     NONE MyType.DoSomething7()
000007ff01a9b328 000007ff01b2e0f0     NONE MyType..ctor()
000007ff01b01280 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01e7a810 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01e7aac0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01e83240 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01f19520 000007ff01b2e178      JIT MyType.RunWorkerCompleted(System.Object, System.ComponentModel.RunWorkerCompletedEventArgs)
000007ff01e7ace0 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01e7b7a0 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01e7b710 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01e7d2b0 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01b015f0 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01b88ce0 0000000000000000      JIT 0000000000000000 is not a MethodDesc
000007ff01a9b3e0 000007ff01b2e200     NONE MyType.DoSomething8()
000007ff01b921e0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01b933b0 0000000000000000     NONE 0000000000000000 is not a MethodDesc
000007ff01b93870 0000000000000000     NONE 0000000000000000 is not a MethodDesc

I'm guessing all the missing entries (those listed with "is not a MethodDesc") are due to this not being a full mini-dump. Is that right?

Gyrfalcon

You shouldn't need symbols to get a managed stack trace. However, you do generally need a full dump. Is this a full dump of the process or just a mini-dump? – Rosierosily

Yes, the missing entries are most likely caused by the dump not being full. – Rosierosily

Answer 1

In a comment above, you mention that the || command yields "User mini dump". In order to properly debug .NET code, you need a full dump, which would indicate "Full memory user mini dump" from the || command. I think this is your problem. Without access to the full loader heaps, it's not possible to map a code address back to a .NET method, so you can't get a stack trace. If you can reproduce this problem, capture a full dump. You can use ADPlus, ProcDump or DebugDiag to capture a dump on crash.

Rosierosily

Answer 2

It looks like WinDbg doesn't pick up symbols for your DLL. You can look into that by setting the symbol path and using !sym noisy to troubleshoot as necessary.

I can't say why !ip2md doesn't work in this case, but there are other ways to get the code for DoSomething2. Try !name2ee on the method name, e.g. !name2ee *!TypeName.DoSomething2 or you can get it via the type itself like this !name2ee *!Namespace.TypeName and then !dumpmt -md <method table> on the method table you get from !name2ee.

Once you have the code, the !u command can show you an .NET annotated dump of the assembly code. By using the offset from the exception, you may be able to determine the nature of the NullReferenceException.

Preparation

Thanks! I thought I did have symbols loaded. Without symbols, I get different output (see my edits in the question). I'll try out the alternative methods soon and report back. – Gyrfalcon

Most of the .NET debugging commands won't work if the dump is not a full dump. I wonder if that's the case here? – Rosierosily

@SteveJohnson, is there a way for me to tell if this is or isn't a full dump? the !ip2md command does work on getting the method descriptor of the to function where the final crash occurred. – Gyrfalcon

Run the || command. I get: 0:000> || . 0 Full memory user mini dump: D:\xyz.DMP – Rosierosily

Okay, I get: . 0 User mini dump: C:\mydump.dmp – Gyrfalcon

@Brian Rasmussen. I tried the !name2ee approach (see Edit 2 in my question). It seems like the information for the method I'm interested is missing. Based on the other comments, I assume this is due to it not being a full minidump. So, I guess at this point I'm out of luck with the dump i have. – Gyrfalcon

@MattSmith Thanks for the update. I does look like something is wrong and it may be that you have an incomplete dump. WinDbg tells you the kind of dump you have when you open the dump. How did you capture the dump? – Preparation

I didn't capture the dump personally. We have a crash reporting system in place which makes the dumps available for a given crash bucket. I'm going to attempt to track down how we gather the dump and see if we can change over to a full dump. Thanks! – Gyrfalcon

Answer 3

You are probably using a stripped PDB file, the default generated for a project in the Release build. All file and line number info is removed from such a file.

Switch to the Release configuration, Project + Properties, Build tab, Advanced, Debug Info = "full".

This is intentional btw, line number info is not very accurate for the Release build. The jitter optimizer moves code around so you'll need to keep in mind that a displayed line number is an approximation.

Kosygin