How to know who kills my threads

Question

I got a thread that is just banishing.. i'd like to know who is killing my thread and why.

It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.

As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.

public void RunWorker()
{
    Thread worker = new Thread(delegate()
    {
        try
        {
            DoSomethingForALongLongTime();
        }
        catch(Exception e)
        {
           //Nothing is never logged :(
           LogException(e);
           throw e;
        }
    });

    worker.IsBackground = true;
    worker.SetApartmentState(System.Threading.ApartmentState.STA);
    worker.Start();
}

EDIT: Addressing answers

• Try/Catch Possible exceptions:
  It's implemented and it catches nothing :(
• Main Thread dying:
  This thread is created by the web server, which continues to run
• Work completion:
  The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.

Having thought of these things brought me to this question, who is killing my threads??

ps. It's not Lady Goldent in the living room with the candle stick :)

Luciolucita

Does an exception occur when the thread is nixed, and if so, what does it say? Are you sure the thread does not simply finish, which would also explain its disappearance. – Dromond

This could make for a good episode of CSI. ;) – Handset

If this ASP.NET and/or IIS, please add those tags – Avitzur

Two comments: * You say "possible" exceptions. Maybe you're missing something so I would suggest catching Exception (for the sake of debugging). * Put a breakpoint at the normal exit point of the thread. See if it hits. These two measures would exclude two possible scenario's with certainty. – Dromond

What is executionTimeout set to? msdn.microsoft.com/en-us/library/e1f13641.aspx – Ceuta

@Luciolucita I use similar code on my web project with out having any problem (and my threads that do work is for indexing and stats calculations and one times they takes them 26 hours to finish). Now I have make a test from your code and I let the thread work more than 5 minutes and everything is start and end - no one kill them. Can you give us a real project that have the problem that you mention ? because I can not find the killer. – Diageotropism

@Hans Default value, 110 secs – Luciolucita

@Diageotropism i cannot provide more information :( On the other hand, is it possible to record in some way the moment of death of the thread? That will be enough for me – Luciolucita

@mcabral, is your web app being restarted by IIS? – Oxymoron

Colonel Panic in the apartment with the pipe. – Lucubrate

Can you run this as part of the Application, and not tied to any specific request, then have it called from a Request? – Cogswell

Answer 1

Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.

I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.

Extra

This looks very promising; it fits all symptoms. I'm surprised though, that IIS can migrate all thread state to a sub-process without his DoStuffForLongTime() function not even being aware. Is it shared memory? – Chiang

because this is background thread it is not have to run forever. once the host's last foreground thread is ended everything else goes south... I cant tell whether you can trick IIS but I'd simply change to foreground and test. – Terpstra

Answer 2

Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.

---

Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.

This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.

If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.

Schismatic

Perhaps your thread did something completely illegal and died to segmentation violation (which wouldn't show up as an exception). You might create a signal handler which raises an error flag, which you could then check. – Chiang

@Justin: he's running .NET. There's nothing so completely illegal that there would be no exception. A "segment violation" cannot occur, unless working with unmanaged code. – Schismatic

Answer 3

A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.

1. Download Debugging Tools for Windows if you don't already have it
2. Run windbg.exe, attach to your process
3. Break into windbg, type sxe et to enable breaking on thread exit
4. When the debugger breaks, inspect the state of the system, other threads, etc.
5. To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)

If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.

Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.

Caswell

If someone is calling Thread.Abort() on your thread, then you might try catching the special exception ThreadAbortException, to know about it. This won't tell you who, but it will tell you when. – Chiang

Answer 4

I don't know the answer, but some thoughts:

• Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
• Are there any points where it exits normally? Try putting some logging on them.
• Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?

UPDATE

You said:

This thread is created by the web server, which continues to run

If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.

Extra

Plausible. A good summary is available here: bluedragon.blog-city.com/… – Ceuta

Answer 5

Your edit reveals the answer:

It's the butler web server.

How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?

Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.

On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.

Avitzur

+1. "LongLong" work should probably be offloaded to a dedicated application/service. – Ament

Answer 6

The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.

Pumphrey

Answer 7

A background thread will only run as long there are foreground threads runnnig.

As soon that all foreground threads end, any background thread still running will aborted.

Mumbletypeg

Answer 8

If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.

Schoolmaster

Answer 9

A simple answer would be: "The killer doesn't leave a name card" ;)

• If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
• If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process

Hope this helps.

Consentient

Answer 10

You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...

Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.

And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?

UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.

Hoofed

Answer 11

When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).

List runningThreads = ...
public void RunWorker() {
    Thread worker = new Thread(delegate()
    ..
    runningThreads.add(worker);
    worker.Start();
}

public void checkThreads() {
 for (Thread t : runningThreads) {
   Console.WriteLine("ThreadState: {0}", t.ThreadState);
 }
}

Chiang

Answer 12

It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.

What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).

Toronto

Answer 13

You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.

Oxymoron

Answer 14

use AsyncTasks to achieve your long running work in asp.net

Caravel

Answer 15

Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx

it may give you some information if you miss some exceptions

Cusick