How to list running tasks in .net memory dump

Question

We have a complex ASP.Net Core application using async/await pattern. The app stopped to respond recently, and we took a memory dump for it. We suspect that there is some async operation that make the app stuck, but not sure which one. After taking memory dump for the web application, we can see very few running threads since the thread is returned to thread pool due to usage of async/await. The question is, is it possible to list running tasks in the memory dump, and where they have run to, so that I can tell which async operation makes the application stuck? For sync blocking calls it is easy - just list the call stacks of all active threads. But for async operation, it doesn't work any more. (Adding more traces is a possible approach but the situation is that we can't garantee we have enough traces for every async operation in the app and it's dependent libraries.)

For example, if an ASP.Net Core app is stuck in some code like this one, how can I tell it from memory dump?

public async Task SomeBadMethodInADependentLibrary()
{
    TaskCompletionSource<int> tcs = new TaskCompletionSource<int>();
    await tcs.Task;
}

Dunsany

Answer 1

You can certainly find task objects in the heap and start analyzing them manually with the SOS commands, e.g. like this beginning of a debugging session:

0:013> !dumpheap -stat -type Task
Statistics:
      MT    Count    TotalSize Class Name
[...]
71e03f28        4          160 System.Threading.Tasks.Task
Total 28 objects

0:013> !dumpheap -mt 71e03f28 
 Address       MT     Size
022bd900 71e03f28       40       
[...]

0:013> !do 022bd900
Name:        System.Threading.Tasks.Task
MethodTable: 71e03f28
EEClass:     719cd6e0
Size:        40(0x28) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
71df1638  40019cd       1c         System.Int32  1 instance        3 m_taskId
71defb44  40019ce        4        System.Object  0 instance 022bd8e0 m_action
[...]

0:013> !DumpObj 022bd8e0
Name:        System.Action
MethodTable: 71e0588c
EEClass:     719357b8
Size:        32(0x20) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
71defb44  40002b5        4        System.Object  0 instance 022bd8e0 _target
71defb44  40002b6        8        System.Object  0 instance 00000000 _methodBase
71df2bdc  40002b7        c        System.IntPtr  1 instance  4b00e64 _methodPtr
71df2bdc  40002b8       10        System.IntPtr  1 instance   4e0c30 _methodPtrAux
[...]

0:013> !u 4e0c30 
Unmanaged code
004e0c30 e833df8372      call    clr!PrecodeFixupThunk (72d1eb68)
[...]

And now it's beginning to become cumbersome...

The most convenient way (in WinDbg) from my point of view is using the !TaskTriage command of Mex (Github):

0:013> !TaskTriage
Normal Mode - not showing successful Tasks
Address    Target     Status                        Method                             Exceptions
==================================================================================================
022bd900 | 022bd8e0 | TASK_STATE_DELEGATE_INVOKED | Demo.Program.printMessage()     |     <none>
022bd974 | 022bd868 | TASK_STATE_DELEGATE_INVOKED | Demo.Program+<>c.<Main>b__0_0() |     <none>
022bd9bc | 022bd868 | TASK_STATE_STARTED          | Demo.Program+<>c.<Main>b__0_1() |     <none>
022bda04 | 022bd868 | TASK_STATE_STARTED          | Demo.Program+<>c.<Main>b__0_2() |     <none>
==================================================================================================
Address    Target     Status                        Method                             Exceptions

The idea of using WinDbg over Visual Studio is good, since both VS2015 and VS2017 will not be able to give the same results from the dump file:

Caning