Best Data Store for huge data with large number of reads and writes

Question

I need to store around 100 millions of records on the database. Around 60-70% of them will be deleted daily and same amount of records are inserted daily. I feel a document database like Hbase, Big Table would fit in this. There are many other data stores like Cassandra, MongoDb, etc. Which data store would be useful for this kind of problem as there will be huge amount of reads/writes(order of 10's of millions) daily.

Precatory

Perhaps you wanted a efficiency comparison. You may see each of these data-stores performances on their websites. We use Cassandra for similar situation, but I haven't played with other datastores – Hummer

the nature of your data probably affects this database comparision as well – Karyosome

a lot of data stores can handle 100 million of records and these read/delete rates. What's the size of the records? what are your data access patterns etc. write vs. read ratios etc.? – Bestir

@ArnonRotem-Gal-Oz: The size of each record could be ~100KB, and read-to-write ratio would be 24:25 on non-weekend and 5:7 on weekends. – Precatory

@Precatory what about accessing the data ? by key? complex queries? – Bestir

@ArnonRotem-Gal-Oz: The access is by key-value pair. The value is actually a json object being stored.. I do not need column orientedness which Hbase and cassandra have. Will be only using Key-value feature.. – Precatory

Answer 1

Based on the characteristics you've mentioned (JSON Documents, accesses by key, 100 million records, balanced read/write) I'd say CouchDB or Membase are good candidates (here's a quick comparison)

Both HBase and Cassandra can probably also work but for HBase you'd need to install a lot of components (Hadoop, ZooKeeper etc) that you won't really use d only use and Cassandra is better when you have more writes than read (at least the last time I used it).

Big Table, is unfortunately internal to google : )

Bestir