ASP.NET Web application doesn't unload AppDomains after deploy

Question

When we deploy our web application we copy all the code to a new directory and then point iis to that new directory. When we do this the number of appdomains increases but never decreases. Also, our Application_End event never seems to fire.

For some unknown time, while both sets of AppDomains are still reported by perfmon, the system performs very poorly while % time in GC spikes to 100%. Eventually I recycle the app pool to get the app running smoothly again.

Extra piece of info: listing the appdomains shows 2 on my dev machine, but 4 run on the live server... we just have one application running in the pool so this means some library we're using is creating app domains.

What should I do to try to debug what's happening? What would prevent an app domain from unloading?

Update 9/3/2014

After getting some more detail log information it looks like the problem is not old app domains staying around, it's new appdomains created during restart. Instead of the application starting one new instance, it starts two. Sometimes we get application_end from the old instance, sometimes we don't.

Update 9/4/2014

Both things are happening. Using process explorer I can see on one of the machines that the old app domain is still there and a new one has started. On the other machine there were only 2 app domains but there was a gap in their sequential ids. So two instances started (we also get a log message from app start), one of them died off almost instantly, leaving 2 app domains.

Mclane

Are you, by any chance using ASP.NET caching with an eviction scheme that reloads the cache when it expires? – Sportscast

We do use ASP.NET application cache. We insert things with a rolling expiration based on the users session timeout. It doesn't reload anything on its own though. – Mclane

What do you mean by "rolling expiration"? Do you mean you update the expiration? What I was asking is whether or not you reload the cache when objects expire based on a callback from the cache manager – Sportscast

If the cache key has not been touched in x number of minutes then the item is removed from cache. The items are not reloaded into cache by the cache manager. The items would only be put back in the cache by a user making a request. – Mclane

Answer 1

Although I am not sure this is the ONLY reason the app was not unloading appdomains, it certainly is one reason. The actual answer is much less interesting than the steps I used to figure it out.

1. I wrote a little ruby script to just make 100s of requests in 10 threads against our app ( setting up jmeter would take more time than this).
2. Run script and made changes to the app config file many times while it ran. This took a couple of minutes.
3. I used process explorer to confirm that there were indeed 4 app domains still loaded into the process.
4. I ran procdump and created a dump file.
5. Loaded the procdump file into visual studio and clicked "Debug with Managed Only" (even though I had no idea if the bug was in managed code or not)
6. Debugger stopped on a line of code where the thread that called Application_End was waiting for a queue to finish processing. By looking at the values of the variables I was able to tell that the queue was no longer going to process items but we were going to wait until the queue was empty.
7. Changed the code and start process over again, this time all the app domains started by my changes to the web config were unloaded.

Mclane

Answer 2

One thing you could check, to help troubleshoot:

Process Explorer actually has a .NET Assemblies tab which lists all the AppDomains loaded by a process. NOTE: The tab only appears for processes that use the .NET Framework.

Enthuse

Thanks, since the code that worked to list app domains fails for some strange reason on the live instance process explorer did allow me to view the AppDomains. At the moment there are 4. SharedDomain,DefaultDomain, and then two instances of what looks like our application. – Mclane

If only I knew what was keeping the app domain up. – Mclane