Is there a heap_stat equivalent for dump analysis of C# applications?

Question

For a previous employer, while performing dump analysis, I have regularly used heap_stat.py, based on PYKD library for Windbg.

I believe heap_stat.py script only works for C/C++ development. Is there an equivalent for C# applications?

Thanks in advance

I've tried to start up !Dumpheap -stat, but loading the SOS extension seems to be a challenge:

Windbg 32-bit:

0:000> .loadby sos clr
0:000> !dumpheap -stat
SOS does not support the current target architecture.

Windbg 64-bit:

0:000> .loadby sos clr
The call to 
LoadLibrary(C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.dll) 
failed, Win32 error 0n193
    "%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.
0:000> .load sos
0:000> !Dumpheap -stat
No export Dumpheap found

Windbg preview:

0:000> .loadby sos clr
The call to 
LoadLibrary(C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.dll) 
failed, Win32 error 0n193
    "%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.
0:000> .load sos
0:000> !Dumpheap -stat
No export Dumpheap found

How can I load the SOS extension and launch that !Dumpheap -stat command?

Marchant

If you have a dump and WinDbg, then the SOS extension and !dumpheap -stat, basically. There's probably also software that tries to make a nice interface out of this, but I have no experience, nor is SO the place for software recommendations. – Psychotechnology

@JeroenMostert: bedankt for the quick reply, but I'm having troubles launching !dumpheap -stat, can you give me one more tip? – Marchant

It looks like you might have created a 64-bit dump of a 32-bit process. This scenario is not supported by SOS. The linked article describes workarounds which may or may not get !dumpheap to work, but it's best to capture a 32-bit dump instead -- procdump will do the right thing by default. Alternatively, uncheck the "prefer 32-bit" checkbox when building your .NET app so it runs as native 64-bit. – Psychotechnology

@JeroenMostert: you are right, and in order to take the 32-bit dump of that 32-bit process, I have launched a 32-bit task manager, which I've found here: %SystemRoot%\SysWOW64\TaskMgr.exe. Now I do have a "dumpheap -stat", which gives some information, but the whole thing looks type-based: for every type or class, there's an entry, while in heap_stat the result was object-based: for every object, there was an entry. That allowed me to see all the collections and their count, revealing interesting information about large collections. Is such a thing possible here? – Marchant

Sure, -stat just summarizes things so you have a high-level view at where the memory goes, but if you really want to get down to the object level that's basically what you get when you leave off -stat. You cannot get things like the size of a managed collection with just !dumpheap, though, as that requires drilling down into object properties. For that you also need things like !DumpObj and possibly some WinDbg scripting. It's been a while since I've had to use SOS to diagnose things, so I couldn't cough up a ready to run script. – Psychotechnology

Note that Visual Studio does offer memory profiling of its own (though it's been a while since I've used that, too); while WinDbg is very satisfying to the inner geek it's not exactly the fastest or most convenient way of diagnosing memory issues unless you already know what you're doing and can quickly zoom in on the problem, the main draws being that it's totally free and also scriptable. If you really want to analyze an entire heap down to the object level with it, though, that's likely going to be very slow. – Psychotechnology

@JeroenMostert: no worries. I've tried the !dumpheap (without stat) and I see a lot of ununderstandable things :-) but those are most probably due to lack of debugging symbols. I'm trying to get that working and the hyperlinks in the !dumpheap results indeed launch !DumpObj commands. I also need to refresh my mind about those things (it's been from 2019 and it was in C++), I'm sure I'll manage. Thanks a lot for your help. – Marchant

@JeroenMostert: you might be surprised, but there's a solution. Smart people somehow patched stuff so that SOS works. See my answer. – Hollinger

Answer 1

The problem

Your crash dump has the wrong bitness. Most likely nowadays that's because you have used Task Manager on a 64 bit system and created a crash dump of a 32 bit application.

Windows comes with two versions of Task Manager: a 64 bit version (C:\Windows\System32\Taskmgr.exe) and a 32 bit version (C:\Windows\SysWOW64\Taskmgr.exe). The path names are somewhat confusing. System32 refers to the native bitness of the OS, which is 64 bit. And WoW64 means Windows 32 bit on Windows 64 bit, which is 32 bit.

Preventing the problem in the future

Have a look at How do I take a good crash dump for .NET? for tools that automatically detect the bitness and create a nice crash dump for you.

Microsoft does still not provide MSCORDAC DLLs (the Microsoft Common Object Runtime Data Access Component) cross-bitness. Therefore your best choice is totake 32 bit crash dumps of 32 bit .NET applications and 64 bit crash dumps of 64 bit .NET applications.

Fixing the problem if you have no other choice

However, there's a fix:

1. Use a 32 bit version of the normal WinDbg (10.0.x.x). WinDbg preview will probably not work, because it creates a child process with the bitness of the dump. You can't load the 32 bit DLL into a 64 bit process.
2. Switch the dump to x86 using .effmach x86
3. Get a copy of soswow64.
4. Load soswow64 using .load and the full path
5. Now load SOS using .loadby sos clr as usual

0:000> .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> .loadby sos clr
0:000:x86> !dumpheap -stat
SOS does not support the current target architecture.

0:000:x86> .unload sos
Unloading C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos extension DLL

0:000:x86> .load C:\Users\T\Downloads\soswow64\soswow64
Successfully hooked IDebugControl::GetExecutingProcessorType.
Successfully patched DbgEng!X86MachineInfo::ConvertCanonContextToTarget.

0:000:x86> .loadby sos clr

0:000:x86> !dumpheap -stat
*** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll
Automatically loaded SOS Extension
CLRDLL: Loaded DLL C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll
c0000005 Exception in C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.dumpheap debugger extension.
      PC: 07261643  VA: 00000000  R/W: 0  Parameter: 00000000


0:000:x86> !dumpheap -stat
Statistics:
      MT    Count    TotalSize Class Name
[...]

Note that the access violation exception for the first SOS command is usual. Just repeat the command.

Hollinger

Sorry, Thomas. I was too fast accepting your answer. When I tried to load the soswow64 DLL, the first message (about hooking IDebugControl) was correct, but the second failed (Failed patching DbgEng!X86MachineInfo::ConvertCanonContextToTarget, stack related commands may not work correctly.). Afterwards the !Dumpheap failed too: (Could not request method table data for object 24263B58 (MethodTable: 7F8E000C).). – Marchant

@Dominique: that's sad. I guess you can only get a different dump then. – Hollinger

I guess you know the developers of soswow64 (or you are one yourself), is there anything I can do in order to help you debug this issue and improve soswow64 accordingly? – Marchant

@Dominique: sorry. I don't know the developer and I'm not involved in the development. One could certainly debug it. WinDbg even has a feature for that: .dbgdbg – Hollinger

.dbgdbg opens a new commandline window, but when I launch the .load C:\...\soswow64 command, I see exactly the same information. – Marchant

.dbgdbg starts a new debugger and attaches to the old debugger so you can debug the debugger and extensions in it. That means, you don't .load soswow64 in the new window, but in the old window. In the new window, you can then break on exceptions, look at the callstack etc. – Hollinger