version control on large files

Question

We happily use SVN for SCM at work. Currently I've got our binary assets in the same SVN repository as our code. SVN supports very large files (it transmits them 'streamily' to keep memory usage sane), but it is SLOOWWWWW.

What asset management software do you recommend, for about a GB (and growing) worth of assets? We would prefer branching and merging (different assets & config files go to different customers).

Ratio

Duplicate of: #275647 – Tectonic

the first paragraph is shared, the questions are completely different. – Ratio

Answer 1

Please be very aware that it is almost impossible to merge binary files! At least automatically. At least I've never heard of a program that supports three-way merging on a binary format. Let alone conflict resolving.

That's why most asset management tools don't have branching, since it makes little sense since you can't merge again. Locking and linear history is better. If you want to "branch", then make a copy of the file with history.

Tsui

Answer 2

At my company, we use Documentum as a ECMS, managing lots (and I mean really lots) of binary files. Documentum (or other ECMS like Alfresco) are supposed to be the "right" solution to manage documents. Documentum supprots tagging and branching, and can expose files as WebDAV (so integration into your workflow can be more or lessa transparent).

That's the theory. In practice, we found Documentum slow, hard to configure and manage. And honestly, even if we have lots of documents, most of them are not larger than a few 100's Mo.

This answer more of what you should not do than what you should do ... sorry ...

Sashenka

Answer 3

Perforce is the only version control system I've heard being used for huge files and whole projects. It's free for two seats, but pretty expensive for more users (around $900 per seat). I've heard it can handle repositories as large as a terabyte.

git might be another option. It behaves somewhat differently from SVN, but is made for bigger projects (e.g. the Linux kernel). I'm not sure if it's good for big binary files.

Lian

With GIT and Adobe Flash projects (45 mb .FLA files), I am begun seeing fatal: Out of memory, malloc failed. I have 2 GB in my machines, so I guess GIT is not good at handling large files. – Tergal

I used Perforce, when I was working in gaming industry. It is sometimes quite annoying, but it handles large files very well. All our assets were kept in perforce. – Ifill

Answer 4

The Mercurial large files extension could help. It adds the ability to tag files as "large". Stores those files in a central store separate from the rest of the repository. Uses hashes to determine which versions of the large files you need and only downloads the ones you need right now.

Fourthly

Answer 5

If you need to manage HUGE files, give a try to Plastic SCM (www.plasticscm.com). AFAIK Perforce is also an option but branching and merging are not that strong.

Communistic