Object storage for a web application

Question

I am currently working on a website where, roughly 40 million documents and images should be served to it's users. I need suggestions on which method is the most suitable for storing content with subject to these requirements.

• System should be highly available, scale-able and durable.
• Files have to be stored permanently and users should be able to modify them.
• Due to client restrictions, 3rd party object storage providers such as Amazon S3 and CDNs are not suitable.
• File size of content can vary from 1 MB to 30 MB. (However about 90% of the files would be less than 2 MB)
• Content retrieval latency is not much of a problem. Therefore indexing or caching is not very important.

I did some research and found out about the following solutions;

• Storing content as BLOBs in databases.
• Using GridFS to chunk and store content.
• Storing content in a file server in directories using a hash and storing the metadata in a database.
• Using a distributed file system such as GlusterFS or HDFS and storing the file metadata in a database.

The website is developed using PHP and Couchbase Community Edition is used as the database.

I would really appreciate any input.

Thank you.

Holotype

That's roughly 120 TB of data, where is this stored now? – Flyover

I calculated that it could be around a 100 TB of data. That's a lot. I would go for a your third or fourth solution. File systems are good at retrieving files, and databases are good at keeping track of those files. Don't put the files themselves in a database, that would be a monstrosity. – Frulla

@Flyover The feature to upload content is not yet implemented. – Holotype

@KIKOSoftware Thanks for the advice. – Holotype

Answer 1

I have been working on a similar system for last two years, the work is still in progress. However, requirements are slightly different from yours: modifications are not possible (I will try to explain why later), file sizes fall in range from several bytes to several megabytes, and, the most important one, the deduplication, which should be implemented both on the document and block levels. If two different users upload the same file to the storage, the only copy of the file should be kept. Also if two different files partially intersect with each other, it's necessary to store the only copy of the common part of these files.

But let's focus on your requirements, so deduplication is not the case. First of all, high availability implies replication. You'll have to store your file in several replicas (typically 2 or 3, but there are techniques to decrease data parity) on independent machines in order to stay alive in case if one of the storage servers in your backend dies. Also, taking into account the estimation of the data amount, it's clear that all your data just won't fit into a single server, so vertical scaling is not possible and you have to consider partitioning. Finally, you need to take into account concurrency control to avoid race conditions when two different clients are trying to write or update the same data simultaneously. This topic is close to the concept of transactions (I don't mean ACID literally, but something close). So, to summarize, these facts mean that you're are actually looking for distributed database designed to store BLOBs.

On of the biggest problems in distributed systems is difficulties with global state of the system. In brief, there are two approaches:

1. Choose leader that will communicate with other peers and maintain global state of the distributed system. This approach provides strong consistency and linearizability guarantees. The main disadvantage is that in this case leader becomes the single point of failure. If leader dies, either some observer must assign leader role to one of the replicas (common case for master-slave replication in RDBMS world), or remaining peers need to elect new one (algorithms like Paxos and Raft are designed to target this issue). Anyway, almost whole incoming system traffic goes through the leader. This leads to the "hot spots" in backend: the situation when CPU and IO costs are unevenly distributed across the system. By the way, Raft-based systems have very low write throughput (check etcd and consul limitations if you are interested).
2. Avoid global state at all. Weaken the guarantees to eventual consistency. Disable the update of files. If someone wants to edit the file, you need to save it as new file. Use the system which is organized as a peer-to-peer network. There is no peer in the cluster that keeps the full track of the system, so there is no single point of failure. This results in high write throughput and nice horizontal scalability.

So now let's discuss the options you've found:

Storing content as BLOBs in databases.

I don't think it's a good option to store files in traditional RDBMS because they provide optimizations for structured data and strong consistency, and you don't need neither of this. Also you'll have difficulties with backups and scaling. People usually don't use RDBMS in this way.

Using GridFS to chunk and store content.

I'm not sure, but it looks like GridFS is built on the top of MongoDB. Again, this is document-oriented database designed to store JSONs, not BLOBs. Also MongoDB had problems with a cluster for many years. MongoDB passed Jepsen tests only in 2017. This may mean that MongoDB cluster is not mature yet. Make performance and stress tests, if you go this way.

Storing content in a file server in directories using a hash and storing the metadata in a database.

This option means that you need to develop object storage on your own. Consider all the problems I've mentioned above.

Using a distributed file system such as GlusterFS or HDFS and storing the file metadata in a database.

I used neither of these solutions, but HDFS looks like overkill, because you get dependent on Hadoop stack. Have no idea about GlusterFS performance. Always consider the design of distributed file systems. If they have some kind of dedicated "metadata" serves, treat it as a single point of failure.

Finally, my thoughts on the solutions that may fit your needs:

1. Elliptics. This object storage is not well-known outside of the russian part of the Internet, but it's mature and stable, and performance is perfect. It was developed at Yandex (russian search engine) and a lot of Yandex services (like Disk, Mail, Music, Picture hosting and so on) are built on the top of it. I used it in previous project, this may take some time for your ops to get into it, but it's worth it, if you're OK with GPL license.
2. Ceph. This is real object storage. It's also open source, but it seems that only Red Hat people know how to deploy and maintain it. So get ready to a vendor lock. Also I heard that it have too complicated settings. Never used in production, so don't know about performance.
3. Minio. This is S3-compatible object storage, under active development at the moment. Never used it in production, but it seems to be well-designed.

You may also check wiki page with the full list of available solutions.

And the last point: I strongly recommend not to use OpenStack Swift (there are lot of reasons why, but first of all, Python is just not good for these purposes).

Inertia

Totally agree there about Ceph. We were looking for a object storage solution and Ceph came up but I just find it too difficult to get started with it. There is a lot of low level mechanism and under-the-hood explanations but not too much of how to set up and use it in production. – Ego

Answer 2

One probably-relevant question, whose answer I do not readily see in your post, is this:

• How often do users actually "modify" the content?

and:

• When and if they do, how painful is it if a particular user is served "stale" content?

---

Personally (and, "categorically speaking"), I prefer to tackle such problems in two stages: (1) identifying the objects to be stored – e.g. using a database as an index; and (2) actually storing them, this being a task that I wish to delegate to "a true file-system, which after all specializes in such things."

A database (it "offhand" seems to me ...) would be a very good way to handle the logical ("as seen by the user") taxonomy of the things which you wish to store, while a distributed filesystem could handle the physical realities of storing the data and actually getting it to where it needs to go, and your application would be in the perfect position to gloss-over all of those messy filesystem details . . .

Accrete