What arguments to use to explain why SQL Server is far better than a flat file

Question

The higher-ups in my company were told by good friends that flat files are the way to go, and we should switch from SQL Server to them for everything we do. We have over 300 servers and hundreds of different databases. From just the few I'm involved with we have > 10 billion records in quite a few of them with upwards of 100k new records a day and who knows how many updates... Me and a couple others need to come up with a response saying why we shouldn't do this. Most of our stuff is ASP.NET with some legacy ASP. We thought that making a simple console app that tests/times the same interactions between a flat file (stored on the network) and SQL over the network doing large inserts, searches, updates etc along with things like network disconnects randomly. This would show them how bad flat files can be, especially when you are dealing with millions of records.

What things should I use in my response? What should I do with my demo code to illustrate this?

My sort list so far:

• Security
• Concurrent access
• Performance with large amounts of data
• Amount of time to do such a massive rewrite/switch and huge $ cost
• Lack of transactions
• PITA to map relational data to flat files
• NTFS doesn't support tons of files in a directory well
• Lack of Adhoc data searching/manipulation
• Enforcing data integrity
• Recovery from network outage
• Client delay while waiting for other clients changes to commit
• Most everybody stopped using flat files for this type of storage long ago for good reason
• Load balancing/replication

I fear that this will be a great post on the Daily WTF someday if I can't stop it now.

Additionally

Does anyone know if anything about HIPPA could be used in this fight? Many of our records are patient records...

Cambridgeshire

Are you sure they weren't advocating a No-SQL approach? That's a popular topic these days. If they are advocating NoSQL, you'd want to address that a little differently. – Bakki

Nope they want nothing but Power Basic and text files. The guy suggesting this hasn't programmed since the early 90's... :( – Cambridgeshire

@jamone: GET THE HELL OUT OF THERE! Seriously - if those "higher ups" have other "friends" who tell 'em who knows what...... – Conduplicate

Yes, I've come to that conclusion and not just based on this. I'm going to see if I can stick it out another couple months to get something better lined up. – Cambridgeshire

Tell your bosses that Dilbert's Know-It-All boss disagrees: dilbert.com/strips/comic/1995-11-17 – Conduplicate

What happened to spur this sudden change? Is there a problem with the current system? Or a feature they wanted couldn't be swiftly delivered? – Selfanalysis

In the 90's doing it with flat files would have been even more heinous than with today's tools and certainly worse than using a 90's (or even 80's) DBMS. – Vinous

And sadly, this exact argument was already presented to me once, and made it to thedailywtf.com: thedailywtf.com/Articles/… – Zoril

Perhaps you could argue that MySQL is middleware layered on top of a flat file system to provide increased efficiency and to make programming easier for the 99% of programmers not as "skilled" as the boss's friend:-) – Gobioid

From what I can tell it appears caused by talk between the two about how much we spend on licensing as well as a large bug we had a few months ago. A bug that was all the fault of an offshore contractor that we were forced to use... I'm also going to combat the licencing cost but that's pretty easy. There are plenty of free alternatives to the software we use that are far better then flat files and Power Basic. – Cambridgeshire

Everyone had good answers, I choose the one with the most up votes, but will be using info from most of these answers. Thanks – Cambridgeshire

Sometimes managers, who just want to cut costs, end up costing the company much more by doing things like using a low quality offshore contractor or implementing "brilliant" ideas like your company's flat file plan. – Percolate

@Cambridgeshire -- what company is this? I'd like to (A) make sure I never apply there, and (B) make sure I'm never a customer of theirs. ` ;) ` – Mortarboard

Keep your files in folders in a file cabnet or in a heap on the floor, both are possible, but which is better? In comparison, what is the cost of the furniture. – Armor

I found one department (outside IT) that was keeping all of their records on CDs in filing cabnets. Over 50k CDs... When IT found out I wrote a site to manage the data on our servers... Wish I could say what company, but I need this job for a little while more. – Cambridgeshire

Everyone in IT that knows about this have already said we are all going to quit if this gets shoved down our throats. – Cambridgeshire

Well the final command just came down from the top, PowerBasic and flat files it is. I quit weeks ago, my last day is tomorrow... – Cambridgeshire

Answer 1

1. Data integrity. First, you can enforce it in a database and cannot in a flat file. Second, you can ensure you have referential integrity between different entities to prevent orphaning rows.

2. Efficiency in storage depending on the nature of the data. If the data is naturally broken into entities, then a database will be more efficient than lots of flat files from the standpoint of the additional code that will need to be written in the case of flat files in order to join data.

3. Native query capabilities. You can query against a database natively whereas you cannot with a flat file. With a flat file you have to load the file into some other environment (e.g. a C# application) and use its capabilities to query against it.

4. Format integrity. The database format is more rigid which means more consistent. A flat file can easily change in a way that the code that reads the flat file(s) will break. The difference is related to #3. In a database, if the schema changes, you can still query against it using native tools. If the flat file format changes, you have to effectively do a search because the code that reads it will likely be broken.

5. "Universal" language. SQL is somewhat ubiquitous where as the structure of the flat file is far more malleable.

Vinous

Answer 2

I'd also mention data corruption. Most modern SQL databases can have the power killed on the server, or have the server instance crash and you won't (shouldn't) loose data. Flat files aren't really that way.

Also I'd mention search times. Perhaps even write a simple flat file database with 1mil entries and show search times vs MS SQL. With indexes you should be able to search a SQL database thousands of times faster.

I'd also be careful how quickly you write off flat files. Id go so far as saying "it's a good idea for many cases, but in our case....". This way you won't sound like you're not listening to the other views. Tact in situations like this is a major thing to consider. They may be horribly wrong, but you have to convince your boss of that.

Zoril

Good point on the tact. I'll make sure to point out that we do use flat files all the time. Error logging, and such. They do have their uses. – Cambridgeshire

+1 for "tact". However you develop your argument - be tactful or it will be written off almost instantly (and you with it). – Malapert

Answer 3

What do they gain from using flat files? The conversion process will be hundreds of hours - hours they pay for. How quickly can flat files generate a positive return on that investment? Provide a rough cost estimate. Translate the technical considerations into money (costs), and it puts the problem in their perspective.

On top of just the data conversion, add in the hidden costs for duplicating a database's capabilities...

• Indexing
• Transaction processing
• Logging
• Access control
• Performance
• Security

Perithecium

If they have hundreds of databases, may I suggest a bigger number than merely "hundreds of hours"? – Harrovian

years, and years more likely. – Cambridgeshire

Excellent point! Presenting all of the hidden costs using actual dollar values will definitely help get the higher-ups to listen. – Percolate

Answer 4

Databases allow you to easily index your data to be able to particular records or groups of records by searching any number of different columns.

With flat files you have to write your own indexing mechanisms. There is no need to do all that work again when the database does it for you already.

Subduct

Because some millionaire who hasn't developed for years told our bosses that's what we have to do... – Cambridgeshire

Answer 5

If you use "text files", you'll need to build an interface on top of it which Microsoft has already done for you and called it SQL Server.

Ask your managers if it makes sense to your company to spend all these resources building a home-made database system (because really that's what it is), or would these resources be better spent focusing on the business.

• Performance: SQL Server is built for storing conveniently searchable data. It has optimized data structures in memory built with searching/inserting/deleting in mind. Usage of the disk is lowered, as data regularly queried is kept in memory.

• Business partners: if you ever plan to do B2B with 3rd party companies, SQL Server has built-in functionality for it called Linked Servers. If you have only a bunch of files, your business partner will give up on you as no data interconnection is possible. Unless you want to re-invent the wheel again, and build an interface for each business partner you have.

• Clustering: you can easily cluster servers in SQL Server for high availability and speed, a lot more than what's possible with text based solution.

Arethaarethusa

Answer 6

You have a nice start to your list. The items I would add include:

1. Data integrity - SQL engines provide built-in mechanisms (relationships, constraints, triggers, etc.) that make it very simple to reduce the amount of "bad" data in your system. You would need to hand code all data constraint separately if you use flat files.
2. Add-Hoc data retrieval - SQL engines, through the use of SELECT statements, provide a means of filtering and summarizing your data with very little code. If you are using flat files, considerably more code is needed to get the same results.

These items can be replicated if you want to take the time to build a data engine, but what would be the point? SQL engines already provide these benefits.

Translucid

Answer 7

I don't think I can even start to list the reasons. I think my head is going to explode. I'll take the risk though to try to help you...

• Simulate a network outage and show what happens to one of the files at that point
• Demo the horrors of a half-committed transaction because text files don't pass the ACID test
• If it's a multi-user application, show how long a client has to wait when 500 connections are all trying to update the same text file
• Try to politely explain why the best approach to making business decisions is to listen to the professionals who you are paying money and who know the domain (in this case, IT) and not your buddy who doesn't have a clue (maybe leave out that last bit)
• Mention the fact that 99% (made up number) of the business world uses relational databases for their important data, not text files and there's probably a reason for that
• Show what happens to your application when someone goes into the text file and types in "haha!" for a column that's supposed to be an integer

Urbanite

Answer 8

Your list is a great start of reasons for sticking with a database.

However, I would recommend that if you're talking to a technical person, to shy away from technical reasons in a recommendation because they might come across as biased.

Here are my 2 points against flat file data storage:

1) Security - HIPPA audits require that patient data remain in a secure environment. The common database systems (Oracle, Microsoft SQL, MySQL) have methods for implementing HIPPA compliant security access. Doing so on a flat-file would be difficult, at best.

Side note: I've also seen medical practices that encrypt the patient name in the database to add extra layers of protection & compliance to ensure even if their DB is compromised that the patient records are not at risk.

2) Reporting - Reporting from any structured database system is simple and common. There are hundreds of thousands of developers that can perform this task. Reporting from flat-files will require an above-average developer. And, because there is no generally accepted method for doing reporting off of a flat-file database, one developer might do things different than another. This could impact the talent pool able to work on a home-grown flat-file system, and ultimately drive costs up by having to support that type of a system.

I hope that helps.

Prodigy

Answer 9

If you are a public company, the shareholders would be well served to know this is being seriously contemplated. "We" all know this is a ridiculous suggestion given the size and scope of your operation. Patient records must be protected, not only from security breaches but from irresponsible exposure to loss - lives may depend up the data. If the Executives care at all about the patients, THIS should be their highest concern.

I worked with IBM 370 mainframes from '74 onwards and the day that DB2 took over from plain old flat files, VSAM and ISAM was a milestone day. Haven't looked back to flat-file storage, except for streaming data, in my 25 years with RDBMSs of 4 flavors.

If I owned stock in "you", dumping it in a hurry the moment the project took off would seem appropriate...

Telemeter

Its privately owned. Otherwise I'd go to the board and I'd expect they would be much easier to convince then just 1 owner. – Cambridgeshire

Even privately owned - if this causes damages for a client by whatever means, the company may / will be hold responsible for gross and willfull neglect. – Iredale

Answer 10

How do you create a relational model with plain text files?

Or are you planning to use a different file for each entity?

Gwendagwendolen

I honestly don't know how I'd map a relational model to text files. I never expected anything like this. – Cambridgeshire

It is a rethorical question. You can't :) But I think this will be your main problem, if you care about the sanity of your data. Even before transactions. – Gwendagwendolen

Of course you "can". Text files can represent anything. At the most basic level, one file per table. I wouldn't dream of ever doing it though... If I remember correctly, Interbase/Firebird has system tables that define everything, including the system tables (a sort of chicken/egg problem). I'm not sure about other databases. – Harrovian

Answer 11

Pro file system:

1. Stable (less lines of code = less bugs, easier to understand, more reliable)
2. Faster with huge data blobs
3. Searching/sorting is somewhat slow (but sort can be faster than SQL's order by)

So you'd chose a filesystem to create log files, for example. Logging into a DB is useless unless you need to do complex analysis of the data.

Pro DB:

2. Transactions (which includes concurrent access)
3. It can search through huge amounts of records (but not through huge blobs of data)
4. Chopping the data in all kinds of ways with queries is easy (well, if you know your SQL and the special "oddities" of your DB)

So if you need to add data rarely but search it often, select parts of it by certain criteria or aggregate values, a DB is for you.

Boru

Blobs luckily aren't used but in 1 case and with that case its such limited scope its a non-issue. – Cambridgeshire

Answer 12

NTFS does not support mass amounts of .txt files well. Depending on how a flat file system is developed, the health of a harddrive can become an issue. A lot of older file systems use mass amount of small .txt files to store data. It's bad design, but tends to happen as a flat file system gets older.

Fragmentation becomes an issue, and you lose a text file here and there, causing you to lose small amounts of data. Health of a hard drive should not be an issue when it comes to database design.

Innerve

Answer 13

This is indeed, on the part of your employer, a MAJOR WTF if he's seriously proposing flat files for everything...

You already know the reasons (oh - add Replication / Load Balancing to your list) - what you need to do now is to convince him of them. My approach on this would two fold.

First of all, I would write a script in whatever tool you currently use to perform a basic operation using SQL, and have it timed. I would then write another script in which you sincerely try to get a flat text solution working, and then highlight the difference in performance. Give him both sets of code so he knows you aren't cheating.

Point out that technology evolves, and that just because someone was successful 20 years ago, this does not automatically entitle them to a credible opinion now.

You might also want to mention the scope for errors in decoding / encoding information in text files, that it would be trivial for someone to steal them, and the costs (justify your estimate) in adapting the current code base to use text files.

I would then ask serious questions of management - foremost amongst them, and I would ask this DIRECTLY, is "Why are you prepared to overrule your technical staff on technical matters" based on one other individual's opinion - especially when said individual is not as familiar with our set up as we are...

I'd also then use the phrase "I do not mean to belittle you, but I seriously feel I have to intervene at this point for the good of the company..."

Another approach - turn the tables - have Mr. Wonderful supply arguments as to why text files are the way forward. You'll then either a) Learn something (not likely), or b) Be in a position to utterly destroy his arguments.

Good luck with this - I feel your pain...

Martin

Oblong

I'd love to turn the tables, however from what I'm hearing he has already convinced the higher ups months ago. But IT is just starting to find out. I think its now up to us to counter... – Cambridgeshire

Put a credible case against it, with universal support from everyone in your company that can actually SPELL "IT", and if you're still overruled, resign. I don't think I'd be able to work in your environment without a belt fed weapon and someone to tell me where the management suite is... – Oblong

Consider the performance trial. We both know that performance is not the most important aspect of this (think ACID), but it is easily demonstrated, and something which ought not to be beyond the wit of even management.... – Oblong

@jamone: I think he has a competing business in his back closet and is looking for a way to make more money by putting a stick in this company's wheels... :) – Harrovian

Answer 14

I suggest you get your retalliation in first, post on Daily WTF now.

As to your question: a business reason would be why does your boss want to rewrite all your systems. From scratch as you would, effectively, have to write your own database system.

For a development reason, you would lose access to the SQL server ecosystem, all the libraries, tools, utilities.

Perhaps the guy that suggested this is actually thinking of going into competition with your company.

Czarevitch

Answer 15

Simplest way to refute this argument - name a fortune 500 company that processes data on this scale using flat files?

Now name a fortune 500 company that doesn't use a relational database...

Case closed.

Geier

Um... Google? Google BigTable isn't a relational database. Of course, it's not exactly a flat file either. And Google isn't necessarily a good example to emulate -- very few companies work with enough volume of data to justify doing things the way Google has done it. – Forestall

Yeah I was aware of Google doing that, but given that they wrote their own OS, and filesystem to make everything work the way it does now I feel like that is easy to explain why they can/did and we can't/shouldn't – Cambridgeshire

Answer 16

Something is really fishy here. For someone to get the terminology right ( "flat file" ) but not know how overwhelmingly stupid an idea that is, it just doesn't add up. I would be willing to be your manager is non-technical, but the person your manager is talking to is. This sounds more like a lost in translation problem.

Are you sure they don't mean no-SQL, as if you are in a document centric environment, moving away from a relational database actually does make sense in some regards, while still having many of the positives of a tradition RDBMS.

So, instead of justifying why SQL is better than flat files, I would invert the problem and ask what problems flat files are meant to solve. I would put odds on money that this is a communication problem.

If its not and your company is actually considering replacing its DB with a home grown flat file system off the recommendation of "a friend", convincing your manager why he is wrong is the least of your worries. Instead, dust off and start circulating your resume.

Exciter

I wish it was a communication error. The "friend" sent a brief explanation of how he did things years ago and now he's filthy rich so it worked. It was pretty clear about using thousands of text files to store data vs using "bloated and expensive database software"... Also they are wanting us to use Power Basic from now on out. 90%+ of what we do is intranet based web apps so there isn't a way to use power basic unless we switch to all client based apps. – Cambridgeshire

Wow, that's horrible. You should send them this thread. And start looking for another job. :) – Halley

Answer 17

•Amount of time to do such a massive rewrite/switch and huge $ cost

It's not just amount of time it is the introduction of new bugs. A re-write of these proportions would cause things that currenty work to break.

I'd suggest a giving him a cost estimate of the hours to do such a rewrite for just one system and then the number of systems that would need to change. Once they have a cost estimate, they will run from this as fast as they can.

Managers like numbers, so do a formal written decision analysis. Compare the two proposals by benefits and risks, side by side with numeric values. When you get to cost 0 to maintain and 100,000,000 to convert they will get the point.

Courtroom

Answer 18

The people that doesn't distinguish between flat files and sql, doesnt understand all arguments that you say before.

---

The explanation must simple as possible, something like this:
SQL is a some kind of search/concurrency wrapper around the flat files.
All the problems that exist currently, will stay even the company going to write the wrapper from zero.

Also you must to give some other way to resolve the current problems, use smart words like advanced BLL or install/uninstall scripting environment. :)

Hirai

Answer 19

You have to speak executive. Without saying it, make them realize they're in way over their heads here. Here's some ammunition:

Database theory is hardcore computer science. We're talking about building a scalable system that can handle millions of records and tolerate disasters without putting everyone out of business.

This is the work of PhD-level specialists. They've been refining the field for a good 20 years now, and the great thing about that is this: it allows us to specialize in building business systems.

If you have to, come right out and say that this just isn't done in the enterprise. It would be costly and the result would be inferior. It's exactly the kind of wheel that developers love to reinvent, and in my opinion the only time you should is if the result is going to be a product or service that you can sell. And it won't be.

Halley