What are the possible causes for IIS to throw a ThreadAbortException and recycle the worker, with IIS logging "IIS configuration change"?

Question

I started seeing errors in a .Net MVC web app hosted on Appharbor whilst a background thread was running - after careful analysis - I can't work out the cause.

Firstly, the exception I noticed is a ThreadAbortException.

However, this is really just signifying that the thread is being killed. Before the thread is killed, you can see a new worker is created by IIS and Application_Start is called on the same machine. Once the new application is up and running, IIS kills the old app and new requests are handled as expected.

At the same time, IIS logs a message of:

ShutDown Message: IIS configuration change
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
ShutDown Stack:    at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()
   at System.Web.Hosting.PipelineRuntime.StopProcessing()

In .Net Health Monitor Logging you get a:

Message: Application is shutting down. Reason: Configuration changed.
Event Detail Code: 50004

A quick google reveals the source code I suspect is the reason for the error:

if (!HostingEnvironment.StopListeningWasCalled && !HostingEnvironment.ShutdownInitiated) {
    // If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a configuration change.
    HttpRuntime.SetShutdownReason(ApplicationShutdownReason.ConfigurationChange, "IIS configuration change");
}

source: https://github.com/Microsoft/referencesource/blob/master/System.Web/Hosting/IPipelineRuntime.cs

My first thought was to check timestamps for file changes, both in the bin folder and the main application directory - however, this error is thrown without any file changes. Given it only happens on Appharbor, I can't attach to the process and debug that way. I've also monitored memory usage, and don't see any issues there.

The source code states:

If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a configuration change.

Hence, what else could be causing the error and application recycle, if the web.config / other config files aren't changing?

Prowler

Do you have access to the windows eventlog ? Have you try asking AppHarbor ? – Drawplate

Yes - asp health monitoring outputs to my DB any events, but simply states: "Application is shutting down. Reason: Configuration changed." - Appharbor say they don't know why it's happening and can't help any further (other than suggest I re-architect my application). – Prowler

health monitoring won't catch all events that can be logged to the windows event log. An external process may do things and may log it on the windows event logs. Furthermore does AppHarbor use a dedicated Application Pool for your application ? – Drawplate

Yep AppHarbor use a dedicated application pool for each application worker. There are no external processes updating the configuration, or related events in the event log. – Desperate

I've got "Configuration changed" shutdowns on the machine with a small amount of memory, and in the end increasing RAM helped. – Pap

Answer 1

It looks like this was a Microsoft bug.

Unexpected ASP.Net application shutdown after many App_Data file changes occur on a server that is running Windows Server 2012 R2

Hotfix: https://support.microsoft.com/en-us/kb/3052480

Last Review: 09/08/2015 16:29:00

Once this hotfix was applied, the errors went away!

Prowler

Answer 2

There are many reasons, which are listed by this helpful blog entry.

• Application pool settings
• The processModel element of machine.config
• Memory limit
• Request limit
• Timeout

Commingle

Do all these reasons cause a "IIS configuration change" message? Or does each one provide a separate, more accurate error / log message? – Prowler

I'm afraid neither this blog entry, nor this answer, actually answers the question. The application is being recycled for none of these reasons according to what I can log, and appharbor's stance. – Prowler

Answer 3

IIS configuration change should happen when something (anything) changes in your IIS application configuration or code. Examples:

• Change in web.confg
• Change in any dll, aspx, etc...

In any of those cases, IIS application will recycle. In addition, IIS will recycle your process every 29 hours by default, but that will probably not be called "IIS configuration change"

Residuum

As per my question - I checked for timestamps of any changes in the bin directory, as well as generally, but can't see anything that stands out. If this was the cause, would there be a way to see which file is causing the recycle? I have placed this monitor code in the application_end, and it is run - the timestamps match the time the code was first deployed. – Prowler

Missed that part in your question. Sorry. However, it is perfectly normal for IIS to recycle your application. It's not being killed but relatively gracefully shutdown, a new process is spawned for new requests and the older process is allowed to finish processing the requests it had. Why is it important to you to find the exact reason for the recycle? You should just assume it's going to happen and work with that. – Residuum

Because it doesn't allow the older process to finish, on appharbor at least, it kills the running process within a specified timeframe (currently 5 minutes). – Prowler

You probably verified this, but still, there is a configuration setting for the Application to recycle every X minutes. Default is 29 hours, but perhaps your app is configured for 5 minutes? Actually, the setting is for Application Pool and not the Application. – Residuum

Yep - it's not due to this. The 5 minutes I mention happens when a new application is deployed - the old worker is killed after 5 minutes. However, this is irrelevant, this recycling happens without any new deploys. My point is the old thread / process is NOT allowed to finish, it is killed by IIS when this event is logged - I can see this as a ThreadAbortException is thrown. This is all explained in the question. – Prowler

You are saying that your thread/process is not allowed to finish. IIS doesn't care/aware that you have background threads running. It will only make sure that the all the requests that were received by the recycled process have finished and than it will kill the process. – Residuum