How to diagnose what is keeping dead threads from being recycled in .NET

Question

A self-hosted C# ASP.NET WEB.API application stops responding to HTTP GET / POST operations in production. I have a memory dump taken from the production server. The only thing unusual in the dmp is a relatively high number of dead threads.

Truncated !Threads output:

0:000> !Threads
ThreadCount:      116
UnstartedThread:  0
BackgroundThread: 5
PendingThread:    0
DeadThread:       110
Hosted Runtime:   no
XXXX   2d       04b87ca0   1019820 Enabled  00000000:00000000 0015e800     0 Ukn (Threadpool Worker)
XXXX   2e       04b87390   1019820 Enabled  00000000:00000000 0015e800     0 Ukn (Threadpool Worker)
XXXX   55       04b86a80   1019820 Enabled  00000000:00000000 0015e800     0 Ukn (Threadpool Worker)
XXXX   53       04b86188   1019820 Enabled  00000000:00000000 0015e800     0 MTA (Threadpool Worker)
XXXX   54       04b858a8   1019820 Enabled  00000000:00000000 0015e800     0 MTA (Threadpool Worker)

The app has 121 System.Threading.Thread objects:

0:000> !dumpheap -stat -type System.Threading.Thread
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
79b59b8c        1           20 System.Threading.ThreadHelper
79b9297c        2           64 System.Threading.ThreadStart
79b9ff90        2          168 System.Threading.ThreadAbortException
79ba0128      121         5808 System.Threading.Thread
Total 126 objects

Looking at a few of the threads, it doesn't appear that there's any references keeping them alive. Example thread:

0:000> !do 051c5578 
Name:        System.Threading.Thread
MethodTable: 79ba0128
EEClass:     798d8ed8
Size:        48(0x30) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79b88b84  400077f        4 ....Contexts.Context  0 instance 00000000 m_Context
79b9b5c4  4000780        8 ....ExecutionContext  0 instance 00000000 m_ExecutionContext
79b9fb08  4000781        c        System.String  0 instance 00000000 m_Name
79b9ffdc  4000782       10      System.Delegate  0 instance 00000000 m_Delegate
79ba6500  4000783       14 ...ation.CultureInfo  0 instance 00000000 m_CurrentCulture
79ba6500  4000784       18 ...ation.CultureInfo  0 instance 00000000 m_CurrentUICulture
79b9f744  4000785       1c        System.Object  0 instance 00000000 m_ThreadStartArg
79b9ab88  4000786       20        System.IntPtr  1 instance  4b77aa0 DONT_USE_InternalThread
79ba2ad4  4000787       24         System.Int32  1 instance        2 m_Priority
79ba2ad4  4000788       28         System.Int32  1 instance        8 m_ManagedThreadId
79b8b878  4000789      18c ...LocalDataStoreMgr  0   shared   static s_LocalDataStoreMgr
    >> Domain:Value  0015e800:NotInit  <<
79b8e434  400078a        c ...alDataStoreHolder  0   shared TLstatic s_LocalDataStore
    >> Thread:Value <<

Checking for GC roots:

0:000> !gcroot 051c5578
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 590
Scan Thread 2 OSTHread 1510
Scan Thread 3 OSTHread c24
Scan Thread 5 OSTHread 14f8
Scan Thread 8 OSTHread 1d48
Scan Thread 9 OSTHread 1e4c

How do I continue the diagnose to find out what is causing this problem?

Vinegarish

This any use for you? msmvps.com/blogs/senthil/archive/2008/05/29/… – Casease

121 threads doesn't hang a web server. This is most likely just a side-effect of whatever else is ailing the program. Something like a dead-locked finalizer thread perhaps. – Harwill

We later discovered that a different service on the same server was hogging the CPU at 100%. Having resolved the greedy service, our problematic service appeared to return to business as usual. That said, I'm keen to understand what was happening here. – Vinegarish

@Jason - My earlier googling came across that post. It was useful but in my case there isn't any obvious roots keeping the Thread objects alive. – Vinegarish