What's the best solution for file storage for a load-balanced ASP.NET app?

Question

We have an ASP.NET file delivery app (internal users upload, external users download) and I'm wondering what the best approach is for distributing files so we don't have a single point of failure by only storing the app's files on one server. We distribute the app's load across multiple front end web servers, meaning for file storage we can't simply store a file locally on the web server.

Our current setup has us pointing at a share on a primary database/file server. Throughout the day we robocopy the contents of the share on the primary server over to the failover. This scneario ensures we have a secondary machine with fairly current data on it but we want to get to the point where we can failover from the primary to the failover and back again without data loss or errors in the front end app. Right now it's a fairly manual process.

Possible solutions include:

• Robocopy. Simple, but it doesn't easily allow you to fail over and back again without multiple jobs running all the time (copying data back and forth)
• Store the file in a BLOB in SQL Server 2005. I think this could be a performance issue, especially with large files.
• Use the FILESTREAM type in SQL Server 2008. We mirror our database so this would seem to be promising. Anyone have any experience with this?
• Microsoft's Distributed File System. Seems like overkill from what I've read since we only have 2 servers to manage.

So how do you normally solve this problem and what is the best solution?

Bandanna

Answer 1

Consider a cloud solution like AWS S3. It's pay for what you use, scalable and has high availability.

Typecast

Answer 2

You need a SAN with RAID. They build these machines for uptime.

This is really an IT question...

Emeritaemeritus

Answer 3

When there are a variety of different application types sharing information via the medium of a central database, storing file content directly into the database would generally be a good idea. But it seems you only have one type in your system design - a web application. If it is just the web servers that ever need to access the files, and no other application interfacing with the database, storage in the file system rather than the database is still a better approach in general. Of course it really depends on the intricate requirements of your system.

If you do not perceive DFS as a viable approach, you may wish to consider Failover clustering of your file server tier, whereby your files are stored in an external shared storage (not an expensive SAN, which I believe is overkill for your case since DFS is already out of your reach) connected between Active and Passive file servers. If the active file server goes down, the passive may take over and continue read/writes to the shared storage. Windows 2008 clustering disk driver has been improved over Windows 2003 for this scenario (as per article), which indicates the requirement of a storage solution supporting SCSI-3 (PR) commands.

Doubletongue

Answer 4

I agree with Omar Al Zabir on high availability web sites:

Do: Use Storage Area Network (SAN)

Why: Performance, scalability, reliability and extensibility. SAN is the ultimate storage solution. SAN is a giant box running hundreds of disks inside it. It has many disk controllers, many data channels, many cache memories. You have ultimate flexibility on RAID configuration, adding as many disks you like in a RAID, sharing disks in multiple RAID configurations and so on. SAN has faster disk controllers, more parallel processing power and more disk cache memory than regular controllers that you put inside a server. So, you get better disk throughput when you use SAN over local disks. You can increase and decrease volumes on-the-fly, while your app is running and using the volume. SAN can automatically mirror disks and upon disk failure, it automatically brings up the mirrors disks and reconfigures the RAID.

Full article is at CodeProject.

Because I don't personally have the budget for a SAN right now, I rely on option 1 (ROBOCOPY) from your post. But the files that I'm saving are not unique and can be recreated automatically if they die for some reason so absolute fault-tolerance is necessary in my case.

Rat

If he finds DFS already an overkill, I am not sure how a SAN can be considered less intensive and costly.... – Doubletongue

Answer 5

I suppose it depends on the type of download volume that you would be seeing. I am storing files in a SQL Server 2005 Image column with great success. We don't see heavy demand for these files, so performance is really not that big of an issue in our particular situation.

One of the benefits of storing the files in the database is that it makes disaster recovery a breeze. It also becomes much easier to manage file permissions as we can manage that on the database.

Windows Server has a File Replication Service that I would not recommend. We have used that for some time and it has caused alot of headaches.

Psychogenic

Answer 6

DFS is probably the easiest solution to setup, although depending on the reliability of your network this can become un-synchronized at times, which requires you to break the link, and re-sync, which is quite painful to be honest.

Given the above, I would be inclined to use a SQL Server storage solution, as this reduces the complexity of your system, rather then increases it.

Do some tests to see if performance will be an issue first.

Raynard