How much data can be stored in MySQL?

Question

I am just a beginner in MySQL, I need to know how much data can be stored in MySQL. I am developing a web crawler, can I store all the data in MySQL, or do I need to use another Database? Which is more faster? What I mean is, which has the highest Writing/Reading Rate? Do I need to reconfigure to add more data?

Pissarro

You mean the entire data of the web? :) – Arthur

You can store as much data in MySQL as you have disk space and your OS can cope with. There's really not a lot of advice anyone can provide, given so vague information. – Iridectomy

@Daniel :yep @MarkJ:why? – Pissarro

@Tom Gullen sure it is(given the OS can account for all the disk space), you might not be able to store it all in one thable though. – Iridectomy

@Nos, you edited your comment and it is now correct. What you said previously however was not! :-) – Klink

A very large amount can be stored in MySQL, but please keep in mind: unless you're very interested in some semantics of HTML-use, don't store actual HTML pages, store the stripped text-content. – Uther

Answer 1

Depends on the operating system.

Operating System             File-size Limit
Win32 w/ FAT/FAT32           2GB/4GB
Win32 w/ NTFS                2TB (possibly larger)
Linux 2.2-Intel 32-bit       2GB (LFS: 4GB)
Linux 2.4+                   4TB
Solaris 9/10                 16TB
MacOS X w/ HFS+              2TB
NetWare w/NSS file system    8TB

http://dev.mysql.com/doc/refman/5.0/en/full-table.html

Your write/read rate is of little concern to you; your bottleneck will be your internet connection.

Klink

i am using a win32/NTFS system,what will i do if the data goes higher than 4tb??its already 3TB – Pissarro

Find a different DBS, SQL server on a 64bit machine can have 16TB. msdn.microsoft.com/en-us/library/ms143432.aspx Just shop around and see what tickles your onion jar. Oracle can handle 8589541376 G on a 'BFT' addressing scheme. I don't think MYSql was designed with your needs in mind. – Klink

@Tom Gullen: So you are suggesting SQL Server,is there any Open Source DB for my needs?what DB did google using? – Pissarro

@Alex: google is using homegrown BigTable (#363456). – Uther

@Uther : Thks for the info,in my case what all DB can use?what are the solutions? what about Hadoop?is it a DB,can i use that for a webcrawler? – Pissarro

.. and the MySQL manual actually suggests some solutions for a bigger table, you could just employ one of those. – Uther

@Wrikken:what are they?can you please say? – Pissarro

@Alex:they are at the link Tom provided. And as far as recommendation: I'd store the crawled files as documents & let sphinx (sphinxsearch.com) index them, I do however not know about its size limits, but you can distribute them over several nodes. – Uther

@Tom : Can i use Hadoop as a DB? – Pissarro

i see a contradiction in the same page forums.mysql.com/read.php?22,379547,381106 says 32 TO per table can you explain – Cochleate

Answer 2

https://forums.mysql.com/read.php?22,379547,381106

InnoDB Size Limits

• Max # of tables: 4 G
• Max size of a table: 32TB
• Columns per table: 1000
• Max row size: n*4 GB
• 8 kB if stored on the same page
• n*4 GB with n BLOBs
• Max key length: 3500
• Maximum tablespace size: 64TB
• Max # of concurrent trxs: 1023

Nanda Kishore Toomula

Sr DBA,Nokia India

CMDBA 5.0

Gentes

what is the max total size please – Cochleate

Answer 3

This comes from their docs. MyISAM permits data and index files to grow up to 256TB by default, but this limit can be changed up to the maximum permissible size of 65,536TB (2567 − 1 bytes).

Paralyze