Architecture recommendation for load-balanced ASP.NET site

Question

UPDATE 2009-05-21

I've been testing the #2 method of using a single network share. It is resulting in some issues with Windows Server 2003 under load:

http://support.microsoft.com/kb/810886

end update

I've received a proposal for an ASP.NET website that works as follows:

Hardware load-balancer -> 4 IIS6 web servers -> SQL Server DB with failover cluster

Here's the problem...

We are choosing where to store the web files (aspx, html, css, images). Two options have been proposed:

1) Create identical copies of the web files on each of the 4 IIS servers.

2) Put a single copy of the web files on a network share accessible by the 4 web servers. The webroots on the 4 IIS servers will be mapped to the single network share.

Which is the better solution? Option 2 obviously is simpler for deployments since it requires copying files to only a single location. However, I wonder if there will be scalability issues since four web servers are all accessing a single set of files. Will IIS cache these files locally? Would it hit the network share on every client request? Also, will access to a network share always be slower than getting a file on a local hard drive? Does the load on the network share become substantially worse if more IIS servers are added?

To give perspective, this is for a web site that currently receives ~20 million hits per month. At recent peak, it was receiving about 200 hits per second.

Please let me know if you have particular experience with such a setup. Thanks for the input.

UPDATE 2009-03-05

To clarify my situation - the "deployments" in this system are far more frequent than a typical web application. The web site is the front end for a back office CMS. Each time content is published in the CMS, new pages (aspx, html, etc) are automatically pushed to the live site. The deployments are basically "on demand". Theoretically, this push could happen several times within a minute or more. So I'm not sure it would be practical to deploy one web server at time. Thoughts?

Varrian

Answer 1

I'd share the load between the 4 servers. It's not that many.

You don't want that single point of contention either when deploying nor that single point of failure in production.

When deploying, you can do them 1 at a time. Your deployment tools should automate this by notifying the load balancer that the server shouldn't be used, deploying the code, any pre-compilation work needed, and finally notifying the load balancer that the server is ready.

We used this strategy in a 200+ web server farm and it worked nicely for deploying without service interruption.

Grassquit

+1 we do the same thing - take a server "out of rotation", deploy, warmup, return to rotation. – Hamrnand

+1 the only way to do it. We called it "in the mix" and "out of the mix" though. – Aggravation

+1 this is a time tried process. Added an answer on how to handle the high update rate. – Bounds

Answer 2

If your main concern is performance, which I assume it is since you're spending all this money on hardware, then it doesn't really make sense to share a network filesystem just for convenience sake. Even if the network drives are extremely high performing, they won't perform as well as native drives.

Deploying your web assets are automated anyway (right?) so doing it in multiples isn't really much of an inconvenience.

If it is more complicated than you're letting on, then maybe something like DeltaCopy would be useful to keep those disks in sync.

Heavyset

As to your latest comment, synchronization software is probably a good solution. DeltaCopy is free, there are other possibly more well suited apps like tgrmn.com/web/file_synchronization.htm and there are plenty really enterprise/expensive ones out there too. – Heavyset

@jayrdub this would cause an asp.net recycle for any non content change i.e. .config, global.asax, dlls. Considering the common load of the site, you need to use a process like the one in Mufaka's answer to have a more stable deployment process. Added an answer for the full scenario-rate of updates. – Bounds

@freddy if .net assets are aspnet_compiler-ed, appdomain cycling probably won't be noticeable. Session dropping would be a problem if they are inproc, but hopefully they aren't since it is load balanced. If the content updates really often, it might not be practical to pull servers in/out of the LB – Heavyset

as far as appdomain cycling, the maxWaitChangeNotification talked about here might help with the timing of things msdn.microsoft.com/en-us/library/e1f13641.aspx helpful: blogs.msdn.com/tess/archive/2006/08/02/… – Heavyset

Answer 3

One reason the central share is bad is because it makes the NIC on the share server the bottleneck for the whole farm and creates a single point of failure.

Aggravation

Yes, which is why you would normally use dual-NIC servers, a switched gigabit network on the inside, caching, and a redundant clustered file server. This sounds like a lot of "Stuff", but the point is, spending $1000 on network to avoid repeated, redundant admin tasks is a good investment. – Sennet

Answer 4

With IIS6 and 7, the scenario of using a network single share across N attached web/app server machines is explicitly supported. MS did a ton of perf testing to make sure this scenario works well. Yes, caching is used. With a dual-NIC server, one for the public internet and one for the private network, you'll get really good performance. The deployment is bulletproof.

It's worth taking the time to benchmark it.

You can also evaluate a ASP.NET Virtual Path Provider, which would allow you to deploy a single ZIP file for the entire app. Or, with a CMS, you could serve content right out of a content database, rather than a filesystem. This presents some really nice options for versioning.

VPP For ZIP via #ZipLib.

VPP for ZIP via DotNetZip.

Sennet

Answer 5

In an ideal high-availability situation, there should be no single point of failure.

That means a single box with the web pages on it is a no-no. Having done HA work for a major Telco, I would initially propose the following:

• Each of the four servers has it's own copy of the data.
• At a quiet time, bring two of the servers off-line (i.e., modify the HA balancer to remove them).
• Update the two off-line servers.
• Modify the HA balancer to start using the two new servers and not the two old servers.
• Test that to ensure correctness.
• Update the two other servers then bring them online.

That's how you can do it without extra hardware. In the anal-retentive world of the Telco I worked for, here's what we would have done:

• We would have had eight servers (at the time, we had more money than you could poke a stick at). When the time came for transition, the four offline servers would be set up with the new data.
• Then the HA balancer would be modified to use the four new servers and stop using the old servers. This made switchover (and, more importantly, switchback if we stuffed up) a very fast and painless process.
• Only when the new servers had been running for a while would we consider the next switchover. Up until that point, the four old servers were kept off-line but ready, just in case.
• To get the same effect with less financial outlay, you could have extra disks rather than whole extra servers. Recovery wouldn't be quite as quick since you'd have to power down a server to put the old disk back in, but it would still be faster than a restore operation.

Breda

Answer 6

Use a deployment tool, with a process that deploys one at a time and the rest of the system keeps working (as Mufaka said). This is a tried process that will work with both content files and any compiled piece of the application (which deploy causes a recycle of the asp.net process).

Regarding the rate of updates this is something you can control. Have the updates go through a queue, and have a single deployment process that controls when to deploy each item. Notice this doesn't mean you process each update separately, as you can grab the current updates in the queue and deploy them together. Further updates will arrive to the queue, and will be picked up once the current set of updates is over.

Update: About the questions in the comment. This is a custom solution based on my experience with heavy/long processes which needs their rate of updates controlled. I haven't had the need to use this approach for deployment scenarios, as for such dynamic content I usually go with a combination of DB and cache at different levels.

The queue doesn't need to hold the full information, it just need to have the appropriate info (ids/paths) that will let your process pass the info to start the publishing process with an external tool. As it is custom code, you can have it join the information to be published, so you don't have to deal with that in the publishing process/tool.

The DB changes would be done during the publishing process, again you just need to know where the info for the required changes is and let the publishing process/tool handle it. Regarding what to use for the queue, the main ones I have used is msmq and a custom implementation with info in sql server. The queue is just there to control the rate of the updates, so you don't need anything specially targeted at deployments.

Update 2: make sure your DB changes are backwards compatible. This is really important, when you are pushing changes live to different servers.

Bounds

For your queue-based deployment solution-- have you ever implemented something like this? Does the queue solution support database updates along with the file updates? Could you provide details? – Varrian

@Varrian added an update about it, short answer: 1) not for deployments (different heavy/long processes), 2) y, but because you don't put all the info in it, just info that you hand to the external publish process/tool 3) added more about this in the update. – Bounds

Answer 7

I was in charge of development for a game website that had 60 million hits a month. The way we did it was option #1. User did have the ability to upload images and such and those were put on a NAS that was shared between the servers. It worked out pretty well. I'm assuming that you are also doing page caching and so on, on the application side of the house. I would also deploy on demand, the new pages to all servers simultaneously.

Monodrama

Answer 8

What you gain on NLB with the 4IIS you loose it with the BottleNeck with the app server.

For scalability I'll recommend the applications on the front end web servers.

Here in my company we are implementing that solution. The .NET app in the front ends and an APP server for Sharepoint + a SQL 2008 Cluster.

Hope it helps!

regards!

Ingather

Answer 9

We have a similar situation to you and our solution is to use a publisher/subscriber model. Our CMS app stores the actual files in a database and notifies a publishing service when a file has been created or updated. This publisher then notifies all the subscribing web applications and they then go and get the file from the database and place it on their file systems.

We have the subscribers set in a config file on the publisher but you could go the whole hog and have the web app do the subscription itself on app startup to make it even easier to manage.

You could use a UNC for the storage, we chose a DB for convenience and portability between or production and test environments (we simply copy the DB back and we have all the live site files as well as the data).

Gallman

Answer 10

A very simple method of deploying to multiple servers (once the nodes are set up correctly) is to use robocopy.

Preferably you'd have a small staging server for testing and then you'd 'robocopy' to all deployment servers (instead of using a network share).

robocopy is included in the MS ResourceKit - use it with the /MIR switch.

Yearling

Answer 11

To give you some food for thought you could look at something like Microsoft's Live Mesh . I'm not saying it's the answer for you but the storage model it uses may be.

With the Mesh you download a small Windows Service onto each Windows machine you want in your Mesh and then nominate folders on your system that are part of the mesh. When you copy a file into a Live Mesh folder - which is the exact same operation as copying to any other foler on your system - the service takes care of syncing that file to all your other participating devices.

As an example I keep all my code source files in a Mesh folder and have them synced between work and home. I don't have to do anything at all to keep them in sync the action of saving a file in VS.Net, notepad or any other app initiates the update.

If you have a web site with frequently changing files that need to go to multiple servers, and presumably mutliple authors for those changes, then you could put the Mesh service on each web server and as authors added, changed or removed files the updates would be pushed automatically. As far as the authors go they would just be saving their files to a normal old folder on their computer.

Tangleberry

The Live Mesh technology is more appropriate for end-user data synchronization. The more appropriate technology for a server environment is something like DFS Replication built into Windows Server 2003 R2 and higher. – Houck

Yeah maybe but then again if you had a lot of distributed users outside the corporate network that need to push updates then DFS is no good. Yes you could put in a VPN for remote users but my point with LiveMesh is that there would be nothing to do. – Tangleberry

Answer 12

Assuming your IIS servers are running Windows Server 2003 R2 or better, definitely look into DFS Replication. Each server has it's own copy of the files which eliminates a shared network bottleneck like many others have warned against. Deployment is as simple as copying your changes to any one of the servers in the replication group (assuming a full mesh topology). Replication takes care of the rest automatically including using remote differential compression to only send the deltas of files that have changed.

Houck

Answer 13

We're pretty happy using 4 web servers each with a local copy of the pages and a SQL Server with a fail over cluster.

Rasorial