How are people solving app pool recycle issues on deployment with large apps?

Question

Currently after a build/deployment of our app (58 projects, large asp.net MVC 3 front end) takes ~15-20secs to load as it goes through the whole 'recycling the app pool' (release configuration).

We do have a web farm if that alters people's answers, but the question really is:

What are people doing in large scale applications where a maintenance window isn't viable (we're a 24/7 very active website) to minimize that initial 'first hit' on the app pool recycle after a deploy?

We've used a number of tools to analyze that startup time and there doesn't really seem to be any way to bring it down so what I'm looking for are what techniques do people employ in order to minimize the impact of a large application deploy affecting users.

Privateer

The upvotes on this question boggle my mind. This question is borderline closeable as "not a real question", and doesn't include much good information. – Grope

...because it's an extremely broad topic. – Grope

it is indeed a broad topic, though I don't think this invalidates it as a question - at this size in asp.net mvc sites there is a problem with app pool recycle, and I'm just asking people how they've solved/mitigated that? – Privateer

My comment is more about the junk upvoting as opposed to the question, really. – Grope

understood :) perhaps it's something that others are facing and hence the usefulness of the question, though I wouldn't like to second guess - i just know I want our users to suffer as little as possible :) – Privateer

There are some answers in an old question of mine #5358520 – Parks

thanks stefan - I hadn't seen that thread, some useful reading in there – Privateer

Answer 1

By default - if you change 15 files in an ASP.NET application at once (even via FTP) then the app pool is automatically recycled. You can change the number of files but as soon as web.config and bin files are changed then it needs to recycle. So in my opinion the ideal solution for an environment like yours would be as follows:

4 web servers (this is an arbitrary number) each server has a status.aspx that the load balancer looks at - use TeamCity to take 2 of these servers "off line" (off the load balancer) and wait 20 seconds for the traffic to filter across. A distributed cache will help keep user experience problems

Use TeamCity to deploy to those 2 servers - run your automated tests etc. and once you are happy put those back into the farm and take the other 2 offline and deploy to those

This can all be scripted / automated. The only issue with this is any schema changes that are not backwards compatible may not allow running the new version site in parallel with old version of the site for the 20 seconds for the load balancer to kick back in

This is good old fashioned Canary Releasing - there are some patterns here http://continuousdelivery.com/patterns/ to help take into consideration. Id also suggest a copy of that continuous delivery book - its like a continuous delivery bible and has got me out of a few situations :)

Clavicorn

Good answer Paul and very much what we had discussed internally. Although the question doesn't address it but I'd be interested to know how you run SQL changes alongside this. – Gradeigh

We tend to make our schema changes backwards compatible - so by versioning our database we can say site x (old version of site) looks for schema version 1 whereas site y (new version of code) targets schema version 2. This way we can run old and new versions. This allows us to release features to only a single set of users if we want to – Clavicorn

Interesting but I can't see how this is feasible in our situation. I think I'll grab you next time I see you about this one. Cheers. – Gradeigh

Answer 2

At the very base you could run a tinyget script against the application after completion of deployment which will "warm up" the application however if a customer hits your site before the script can run, they will still face a delay. What do you currently have in place, what post deployment steps do you have in place?

In a farm environment you could stage deployments too, so take one server out of load balance, update it and then bring that online after deployment and take the other out, complete the deployment and then reintroduce into the farm. How is your SQL Server setup - clustered?

Schoonover

this is all for new architecture - at the moment majority is classic ASP, so no hit at all. So this is 'to come' really and we're just looking at best practice. We've entertained the 'take web1 out of farm, deploy, warmup, bring back into farm' and then rinse and repeat for rest of farm and it may be this is the only elegant solution - I assumed that with so many large scale sites using asp.net derivatives now that someone would have solved this elegantly (I'll try to see what SO do) – Privateer

I've been looking at Web Farm Framework and Web Deploy to look at configurations like this but haven't had enough time to fully work through a large number of scenarios. How many deployments are you doing/envisage daily? If multiple can they be combined or do you need to be that reactive? – Schoonover

they're very much reactive unfortunately - and at least 1 per day during busy times, at most? I've seen a day with 10 deployments, though all small so could likely be grouped into 4-5 larger – Privateer

@AndrewWestgarth the good thing about canary releasing is that you have the time to run a warm up script so that will help out too :) – Clavicorn

Answer 3

copy and paste from my post here

We operate a Blue/Green deployment strategy on a 4 tier architecture which has a web site over 4 servers at the top tier. Due to the complexity the architecture introduced for deployments, we needed a way to deploy without disturbing any traffic to the "live" site. Following Fowler's advice, but not quite in the same way, we came up with a solution that means we have 2 sites on each server (a blue and a green, or in our case site A and site B). The live site has the appropriate host header, and once we have deployed and tested to the non-live site, we then flip the headers of the 2 sites so that what was once live is now the non-live site, and vice-versa. The effect is, a robust deployment that can be done in business hours and with the highest level of confidence.

This of course complicates your configuration and deployment slightly, but it's worth the effort. I guess it kind of goes without saying that you want to script both the deployment, and the host header swapping.

Tailrace

Answer 4

Firstly, unless you're running Google or something bigger, does a 15-20s load time at 3am for a handful of users really impact that much? I'd say the effort invested in eliminating the occasional lag would far outweigh the 15-20s inconvenience of a couple of users.

I consider it a necessary evil of using ASP.NET unfortunately. Using a pre-compiled site (.DLLs instead of the code-behind files) will lessen the time but not necessarily eliminate it.

The best thing you can do is use something like a status notification bar to warn users they may experience some "issues" during "essential maintenance".
But even then, I'd say in terms of user experience it'd be better to keep quiet and have a handful of people blame their "slow internet" when your site takes 20s to load on one occasion, than announce to all and sundry that it will be slow.

M16

that'd be nice if we could rely upon that maintenance window and I agree, it's less of an issue. We unfortunately have deployments sometimes 4-5 times per day when the site is busy :/ – Privateer

If a site takes 15-20s to load, I go somewhere else (unless I can't). That's an unacceptable amount of delay for almost all sites. – Grope

aye - sometimes very hectic here, though at present with our classic asp site it's obviously not a problem (deploy only that which has changed) - it's when with the new architecture we have to deploy DLLs that I panic – Privateer

@AndrewBarber ...but if a site's down for maintenance, you sit and wait for it to come back? – M16

If it's been 15s to load, I assume it's down for maintenance or just plain down, and go somewhere else. – Grope

@AndrewBarber I don't see how that's worse than being explicitly told it's down and going somewhere else? Tell everyone there's maintenance happening and they'll all assume it's not working. Silently cause a 15s delay and only the first couple of hits will be affected - some of whom may go elsewhere. – M16

@M16 - What about sites that are doing continuous deployment? 4-5 isn't a great number at all. – Gradeigh

@M16 I think 4 - 5 is a good number - just because something is tough to do and requires a good structure does not mean it should be avoided. Etsy are doing up to 52 deployments to PRODUCTION a day. This is something I personally am in awe of! – Clavicorn

@Clavicorn It's just a pattern that doesn't sit well with me. If you're doing 52 deployments per day, that's called development and should be done on a dev server. If the 52 changes are critical enough to be released immediately, then you have gone live FAR too early. If they're just improvements, they can be done in one step at the end of the day. – M16

@M16 disagree im afraid - I believe in getting software done and released as fast as possible - I work in a very Lean environment and one of the principles of Lean is to reduce waste. Continuous deployment helps reduce time awaiting around pre deployment. Waiting time is waste IMO – Clavicorn

@Clavicorn I understand the philosophy, but does "as fast as possible" really mean there's any value in deploying a minor update a few hours sooner than you otherwise would have? I'd say no - unless it's a KILLER feature! – M16

@M16 but doesnt releasing something to a customer class as adding value? Look at Google Chrome - that does many updates a day and its for the benefit of the product. Compare that to IE, where releases are very infrequent, the customer gets a tough upgrade experience and also doesnt have value delivered for a long time – Clavicorn

@M16 oh and any feature that will be delivered to a user is a KILLER feature! – Clavicorn

@Clavicorn I think Chrome and IE are two extreme examples, neither of which are ideal/necessary. As a user, I don't know when a new feature is available until I'm given access to it. Whether I get 24 updates at midnight rather than 1 update every hour simply isn't going to affect me in any significant way. It's just not that important. I'm not advocating a new version every year with a bazillion updates (IE) - just saying waiting a few more hours to deploy minor changes isn't going to adversely affect anyone. – M16

Just because there is a 15s delay for app restart at 3am, doesn't mean that there won't be a 15s delay when the app pool recycles in the middle of the day, unexpectedly. It is wise to handle the delay in a sensible manner at all times, then you can even deploy at 3pm, instead of 3am. I know this post is ancient, but I would have mentioned this even then; it bears mentioning for people who stumble upon this question. Unless you work hard to avoid it, IIS app pools periodically recycle automatically. – Laban

Answer 5

You can also try this approach : http://weblogs.asp.net/scottgu/archive/2009/09/15/auto-start-asp-net-applications-vs-2010-and-net-4-0-series.aspx

Judah

Answer 6

without knowing anything about your site, my first thought is that you might be able to break it down into smaller sites so that they start faster individually.

second, with your web farm, i assume you have some sort of load balancing device in front of that from which you can pull machines out of the pool when they are being deployed. don't put them back in the pool until after you have sent a request against the site to get it started up. you should be able to script this such that you are pretty much clicking a button that takes a machine out, deploys to it, and sends a request after it's back up and happy.

Stoops

indeed so re: the webfarm, and so far it seems like that is the popular suggestion - upvoted thanks :) – Privateer

Answer 7

You can consider using aspnet_compiler.exe to precompile your application, because I think the delay after deployment is caused by the compilation phase rather than "whole recycling the app pool".

Warton

I believe my asp.net MVC app is compiled, though a lot of an app pool recycle is bringing that app into the app pool so that it can be effectively/quickly served. Some of the time taken is compilation, though for the most part I've not found aspnet_compiler to help significantly with anything other than old style asp.net websites (not applications) – Privateer