Distributed File Systems: GridFS vs. GlusterFS vs Ceph vs HekaFS Benchmarks [closed]

Question

I am currently searching for a good distributed file system.

It should:

• be open-source
• be horizontally scalable (replication and sharding)
• have no single point of failure
• have a relatively small footprint

Here are the four most promising candidates in my opinion:

• GridFS (based on MongoDB)
• GlusterFS
• Ceph
• HekaFS

The filesystem will be used mainly for media files (images and audio). There are very small as well as medium sized files (1 KB - 10 MB). The amount of files should be around several millions.

Are there any benchmarks regarding performance, CPU-load, memory-consumption and scalability? What are your experiences using these or other distributed filesystems?

Lemke

Isn't this a shopping question? – Bouton

no, it's about choosing a good opensource tool – Lemke

Use weed-fs (or seaweed-fs now it is called). It's built to store these kind of files, started as an implementation of facebook's haystack paper. github.com/chrislusf/weed-fs – Dyson

I was going to choose seaweedfs until I noticed that it's not being actively developed anymore and the fact that it hasn't reached 1.0 – Studbook

Good Question, sometime, Good questions are closed in SE. It is one of the problem with SE. – Dicarlo

Answer 1

I'm not sure your list is quite correct. It depends on what you mean by a file system.

If you mean a file system that is mountable in an operating system and usable by any application that reads and writes files using POSIX calls, then GridFS doesn't really qualify. It is just how MongoDB stores BSON-formatted objects. It is an Object system rather than a File system.

There is a project to make GridFS mountable, but it is a little weird because GridFS doesn't have concepts for things like hierarchical directories, although paths are allowed. Also, I'm not sure how distributed writes on gridfs-fuse would be.

GlusterFS and Ceph are comparable and are distributed, replicable mountable file systems. You can read a comparison between the two here (and followup update of comparison), although keep in mind that the benchmarks are done by someone who is a little biased. You can also watch this debate on the topic.

As for HekaFS, it is GlusterFS that is set up for cloud computing, adding encryption and multitenancy as well as an administrative UI.

Sextans

thanks for your great explanations and comments. you are right, i was not quite correct with the term file system. i am ok with something object system like as long as there are ways to deliver the files directly from a webserber. that's where nginx-gridfs comes into play. works great for me. i have the feeling that glusterfs and ceph are much more difficult to setup and configure – Lemke

Note that ceph has several aspects: rados is the underlying object-storage, quite solid and libraries for most languages; radosgw is an S3/Swift compatible system; rbd is a shared-block-storage (similar to iSCSI, supported by KVM, OpenStack, and others); CephFS is the POSIX-compliant mountable filesystem. – Illimani

Answer 2

After working with Ceph for 11 months I came to conclusion that it utterly sucks so I suggest to avoid it. I tried XtreemFS, RozoFS and QuantcastFS but found them not good enough either.

I wholeheartedly recommend LizardFS which is a fork of now proprietary MooseFS. LizardFS features data integrity, monitoring and superior performance with very few dependencies.

---

2019 update: situation has changed and LizardFS is not actively maintained any more.
MooseFS is stronger than ever and free from most LizardFS bugs. MooseFS is well maintained and it is faster than LizardFS.

RozoFS has matured and maybe worth a try.
GfarmFS have its niche but today I would have chosen MooseFS for most applications.

Monied

wow, so many names i never heard of – Lemke

GfarmFS is another good file system. – Monied

I have used XtreemFS in production on 20 servers where each had 5x2TB hard drives. I can say that I could not be be happier with it. Why you particularly do not like it? – Thrombokinase

Because I tested XtreemFS and found that it does not work well. There are problems like data corruption (#359), read errors in degraded mode (#357/#235), crippled read-only mode (#358) etc.; build system is a mess plus XtreemFS depends on old (not updated since 2007) non-free JAR (#309, #173) so XtreemFS is in violation of DFSG and not distributable in Debian. Also I'm not happy about how devs respond to bugs. Finally XtreemFS is written in poor language notorious for inefficient memory management so naturally XtreemFS can't stand against GfarmFS and LizardFS in performance comparison... – Monied

Answer 3

OrangeFS, anyone?

I am looking for a HPC DFS and found this discussion here: http://forums.gentoo.org/viewtopic-t-901744-start-0.html

Lots of good data and comparisons :)

After some talk the OP decided for OrangeFS, quoting: "OrangeFS. It does not support quotas nor file locks (though all i/o operations are atomic and this way consistency is kept without locks). But it works, and works well and stable. Furthermore this is not a general file storage oriented system, but HPC dedicated one, targeted on parallel I/O including ROMIO support. All test were done for stripe data distribution. a) No quotas — to hell quotas. I gave up on them anyway, even glusterfs supports not common uid/gid based quotas, but directory size limitations, more like LVM works. b) Multiple active metadata servers are supported and stable. Compared to dedicated metadata storage (single node) this gives +50% performance on small files and no significant difference on large ones. c) Excellent performance on large data chunks (dd bs=1M). It is limited by a sum of local hard drive (do not forget each node participates as a data server as well) speed and available network bandwidth. CPU consumption on such load is decent and is about 50% of single core on a client node and about 10% percents on each other data server nodes. d) Fair performance on large sets of small files. For the test I untared linux kernel 3.1. It took 5 minutes over OrangeFS (with tuned parameters) and almost 2 minutes over NFSv4 (tuned as well) for comparison. CPU load is about 50% of single core (of course, it is actually distributed between cores) on the client and about several percents on each node. e) Support of ROMIO MPI I/O API. This is a sweet yummy for MPI aware applications, which allows to use PVFS2/OrangeFS parallel input-output features directly from applications. f) No support for special files (sockets, fifo, block devices). Thus can't be safely used as /home and I use NFSv4 for that task providing users quota-restricted small home space. Though most distributed filesystems don't support special files anyway. "

Treasonable

OrangeFS is interesting but... It FTBFS (v2.9.0); it has no data integrity checks yet (to be implemented); its bug tracker is closed so I can't submit a bug report... :( – Monied

too late for you, i'm sure, but OrangeFS (and PVFS before it) are old school. shoot an email to the mailing list to report a bug. – Cauca

Btw... I was testing FhGFS (FraunhoferFS) and decided to use it. Not very much publicized but very solid. – Treasonable

Answer 4

I do not know about the other systems you posted but I have made a comparison of 3 PHP CMS/Frameworks on local storage vs GlusterFS to see if it does better on real world tests than raw benchmarks. Sadly not.

http://blog.lavoie.sl/2013/12/glusterfs-performance-on-different-frameworks.html

Yesman