MongoDB limit storage size?

Question

What is MongoDB's storage size limit on 64bit platforms? Can MongoDB store 500-900 Gb of data within one instance (node)? What was the largest amount of data you've stored in MongoDB, and what was your experience?

Panegyric

we're doing 12 TB with PSA failover and it's insane to backup/restore but it runs. – Kara

Answer 1

The "production deployments" page on MongoDB's site may be of interest to you. Lots of presentations listed with infrastructure information. For example:

http://blog.wordnik.com/12-months-with-mongodb says they're storing 3 TB per node.

Eurydice

Answer 2

The MongoDB's storage limit on different operating systems are tabulated below as per the MongoDB 3.0 MMAPv1 storage engine limits.

The MMAPv1 storage engine limits each database to no more than 16000 data files. This means that a single MMAPv1 database has a maximum size of 32TB. Setting the storage.mmapv1.smallFiles option reduces this limit to 8TB.

Using the MMAPv1 storage engine, a single mongod instance cannot manage a data set that exceeds maximum virtual memory address space provided by the underlying operating system.

                            Virtual Memory Limitations

Operating System           Journaled                Not Journaled

   Linux                 64 terabytes               128 terabytes

Windows Server 2012 R2
and Windows 8.1          64 terabytes               128 terabytes

Windows (otherwise)       4 terabytes                8 terabytes

Reference: MongoDB Database Limit.

Note:The WiredTiger storage engine is not subject to this limitation.

Another way to have more than 2GB on a single node is to run multiple mongod processes. So sharding is one option or doing some manual partitioning across processes.

Hope This helps.

Agrimony

I am curious. What exactly means "each database"? Is a single installation on a server considered a database or an entire cluster with multiple shards? (Probably NOT the latter - right?, otherwise the absolute max of a mongo-cluster would be 32TB and that'd be ridiculous) – Horne

I have the same query as @Horne after this discussion. I am curious to know Mongodb limitations. If anyone has come across a node computation example - cluster with C nodes and Y Memory and then the storage would be C*0.8*Y something like that, could you please share a link. – Interlinear

Answer 3

You won't run anywhere near hitting the cap with 1TB on 64 bit systems, however Mongo does store the indexes in memory so a smooth experience depends on your index size and how much memory you have. But if you have a beefy enough system it won't be a problem.

Voelker

In our case we're crunching numbers on years of historical data retrieved from certain markets of eBay. In our situation, 1TB is most definitely a concern of ours- Don't be too quick to rule out such large numbers. But for every day use you are correct- 1TB is pretty extreme. – Hackler

@Joshua Burns I think you misunderstood the answer. I believe @Donnie H meant that 1 TB of data doesn't approach the maximum capacity of MongoDB, not that 1 TB is a volume of data that will never be reached. – Eurydice