Lustre, Gluster or MogileFS?? for video storage, encoding and streaming [closed]

Question

So many options and so little time to test them all... I wonder if someone has experiences with distributed file systems for video streaming and storage/encoding.

I have a lot of huge video files (50GB to 250GB) that I need to store somewhere, be able to encode them to mp4 and stream them from several Adobe FMS servers. The only way to handle all this is with a distributed file system but now the question is which one??

My research so far tells me:

• Lustre: mature proven solution, used by a lot of big companies, best with >10G files is a kernel driver.
• Gluster: new, less mature, FUSE based that means easy to install but maybe slower due to FUSE overhead. Better to handle a large number of smaller files ~1GB
• MogileFS: seems to be only for small files ~MB, uses HTTP for access?? possible FUSE binding in the future.

So far Lustre seems the winner but I would like to hear real experiences for the particular application I have.

Also Hadoop, Redhat GFS, Coda and Windows DFS sound as options so any experiences are welcome. If someone has benchmarks please share.

After some real experience this is what I have learned:

• Luster:
  • Performance: Amazingly fast! I can assert that Lustre can serve a lot of streams and that encoding speed is not affected by accessing files via Lustre.
  • POXIS compatibility: Very good!. No need to modify applications to use luster.
  • Replication, Load Balancing and Fail Over: Very bad!. For replication load balancing we and fail over we need to rely on other software such as virtual IPs and DRDB.
  • Installation: The worst!. Impossible to install by mere mortals. Requires a very specific combination of kernel, lustre patches and tweaks to get it working. And current luster patches usually work with old kernels that are incompatible with new hardware/software.
• MogileFS:
  • Performance: Good for small files but not usable for medium to large files. This is mostly due to HTTP overhead since all files are send/receive via HTTP requests that encode all data in base64 adding a 33% overhead to each file.
  • POXIX compatibility is non existent. All applications require to be modified to use mogilefs that renders it useless for streaming/encoding since most streaming servers and encoding tools do not understand MogileFS protocol.
  • Replication and failover out of the box and load balancing can be implemented in the application by accessing more than one tracker at a time.
  • Installation is relatively easy and ready to use packages exist in most distributions. The only difficulty I found was setting the database master-slave to eliminate the single point of failure.
    • Gluster:
  • Performance: Very bad for streaming. I cannot reach more than a few Mbps in a 10Gbps network. Clients and Server CPU skyrockets on heavy writes. For encoding works because the CPU is saturated before the network and I/O.
  • POXIS: Almost compatible. The tools I use can access gluster mounts as normal folders in disk but in some edge cases things start causing problems. Check gluster mailing lists and you will see there are a lot of problems.
  • Replication, Failover and Load balancing: The best! if they actually worked. Gluster is very new and it has a lot of bugs and performance problems.
  • Installation is too easy. The management command line is amazing and setting replicated, striped and distributed volumes among several servers can not be any easier.

Final conclusion:

Unfortunately the conclusion is "No single silver bullet".

Currently we have our media files in Gluster3.2 in a replicated volume for storage and transcoding. As long as you don't have a lot of servers, avoid geo-replication and stripe volumes things work ok.

When we are going to stream the media files we copy them to a lustre volume that is replicated to a second lustre volume via DR:DB. The wowza server then read the media files from the lustre volumes.

And finally we use MogileFS to serve the thumbnails in our web application servers.

Spallation

this really isn't a programming question and doesn't really belong here. But... I would recommend some form of file synchronization software like, unison, ifolder, or rsync. as the files aren't so huge they could sit on all the servers. All the clustering file systems aren't quiet there in my humble opinion. – Ammonal

I think it's called Lustre... – Planchet

Gluster has a lot of problems with replication. It is buggy. Volume healing doesn't work properly as well. – Cordate

Performance: Good for small files but not usable for medium to large files. This is mostly due to HTTP overhead since all files are send/receive via HTTP requests that encode all data in base64 adding a 33% overhead to each file. It's possible to encode files in HTTP without base64 and I think HTTP servers rarely use base64. Maybe you measured wrong. Most probably the bottleneck is on the use of HTTP headers on every request, which is okay for large files, but awful for small ones. JSON-RPC 2.0 even added the controversial batch requests feature to circumvent this problem. – Trott

Answer 1

GlusterFS improved themselves a lot up to this date. They are now providing "granular locking" for large files. See here: http://www.gluster.org/community/documentation/index.php/WhatsNew3.3 Also it is quite dependent video frame rates you should work for too. If you will not go up to 4K rates, Gluster can solve the storage problems. If there is a huge demand for speed, therefore Infiniband can come in to play.

Walachia

Answer 2

Check out Hadoop Filesystem (HDFS). Its focus is on very large files and parallel task computing (with map/reduce), it has a high latency but very high throughput. It is currently used on such large installations as Facebook and amazon.com

Bluefish

Answer 3

MogileFS is great for that sort of thing. The client libraries varies a bit in quality, but I'd be surprised if there weren't large-ish scale production sites using just about any language to access it.

HTTP is a good protocol for this stuff actually. Who doesn't have a feature-rich and efficient HTTP client?

Cantankerous

Answer 4

From the named systems the most suitable is MoglieFS.

But perhaps you can get by w/out any special system at all. Say you have 4 AdobeFMS servers:

{video0.exmple.com,video1.exmple.com,video2.exmple.com,video3.exmple.com}.

You can distribute all your videos among those 4 servers using simple scheme, like

    /*
     *  pseudo code
     */

    $server_id = get_server_id(filename);
    ...
    ...
    int function get_server_id(filename) 
    { 
       return hash(filename) mod 4;
    }

after you encode videos, your app would

$server_id = get_server_id(file_name)
copy file_name to /mnt/$server_id/

clients will access videos using something like http://videoN.example.com/filename.mp4, where N is calculated from filename using get_server_id() .

Luster/Gluster is really not what you should be looking for. Luster FS is more mature, but developers ask you to treat files on such FS as "cache", i.e. they can be lost at any time.

Luster/Gluster are targeted for use in HPC to allow fast access for huge amounts of data w/out single storage server being performance bottle-neck. Another point for those systems is that they are POSIX-complaint. In HPC/Scientific research environment you usually do not have a time to waist for rewriting your apps because you installed new cool and fast FS.

Besmirch

That works great until one of the servers crashes. Oops. – Skean

the "Oops" on server crash will happen with in all cases, unless special case is taken (even in case of MoglieFS) Also what "Oops" means will be different, in the setup mentioned above 1 server (out of 4) failure means that roughly 1/4 of read/write requests will fail until the failed server restored from backup. if that's not acceptable, then it is relatively easy setup [read-only] replica servers using, for instance, rsync – Besmirch

Answer 5

Map-reduce doesn't help in write/read ratio of 90/10! The constant file size is a good thing and the files are small. So, MogileFS sounds to be good alternative as Luster/Gluster - cache situation is not appropriate.

Prolusion