Git style backup of binary files

Question

I'm in process of integrating GIT version control system into 3ds max to control .max file versioning.

I tested git with max files for several weeks and have found that it suites my needs, but, anyway, git is little bit overcomplicated for this kind of tasks. What I'm looking for is the program that works just like (well, almost) git, but:

1. is naturally born windows program - git failed several times on my network shared folders and destroyed my binary max files. This is the main problem, and I don't know how to solve it.
2. doesn't (doesn't even try to) process file content. I think that speeds up the workflow.
3. has interface close to TortoiseGIT.

What this software has to have is:

1. commit and log commands
2. branching support
3. be distributed and not server oriented like Perforce.

I tested Mercurial, but it is really slow on large binary files. If someone knows how to speed it up (by disabling content management, I suppose) and have TortoiseHg fully functional, any information would be greatly appreciated.

Thank you!

Eleonoraeleonore

Answer 1

Binary files have sort of an ambivalent relationship with VCS because:

• they do not merge (so the all "branching support" is not that interesting)
• they do not diff very well (meaning the storage of their history is not very compact)
• they usually can be generated
• or they can be fetched from other kind of repositories

Binary files are part of Git limits as they tend to slow down the all workflow, and do not scale in size (the repo quickly becomes too large to manage/backup efficiently).

So the question is: could you store those files is some other repo than a VCS?
A Maven repo perhaps (managed by a repo manager like Nexus, not exactly distributed, but made to reference precise version of any kind of set of files).

---

Jakub Narębski (actual Git contributor) rightly points to a Git fork project able to manage more efficiently big files.

[git-bigfiles (ok, the project logo is awesome ;) )

The question remains: is a VCS the right tool for managing such large objects?
Because in my experience, the question of "cleaning up the history" will be asked one day or another, because of a ever increasing amount of disk space used.
And VCS are fundamentally not made for "clean-up" their history.
Git itself cannot do it without changing its SHA1 key, making any future publication to other public repo problematic.

Thordis

Branching is very useful with binaries. Not for subsequent merging but for splitting the versions of a project and swithing between them. Thank you for pointing Marven/Nexus. I'll take a look at Nexus docs. – Eleonoraeleonore

Take a look at git-bigfiles project, if you want version control for large binary files. – Herries

@Jakub: Thank you. I have updated my answer to include that reference, but also have completed my question about the pertinence of using a VCS (distributed or not) for managing large file. – Thordis

Thank you, Jakub! Is it possible to use git-bigfiles with ToroiseGit? – Eleonoraeleonore

Answer 2

What I don't get is why you want to use a DVCS without being able to diff and so to benefit from it, like the first comment says.

I would advice two possible directions:

• have a look at archiva and/or ivy (java tools) which were designed to manage the distribution of binary files, such as jar files, to developers, but it might be too much java-centric (or even development centric) for your use case, or

• try to use rync and its backup possibilities (rsync can save as many versions as you want instead of replacing the previous copy with the new one). It seems well suited to your needs: works well on windows, handles very well binaries, as well as text files, is very bandwidth efficient (can compress), and handles backup with several versions. The only think missing for you would be a GUI, ideally integrated with winexplorer. Maybe there exist some, I don't know..

Hope it'll help.

Cheers,
Christophe.

= Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it. --Linus Torvalds =

Anthony

What makes DVCS better choice is that user is not forced to use some strict server location, he/she can freely store all files on local machine with shared folder. Other user can access that shared folders, clone repo and continue to work on the project. Our approach doesn't involve merging, only branching (forking). Thank you for suggestions. I'll have to look at rsync, may be it's possible to script it to do the job right. – Eleonoraeleonore

The main advantage of (D)VCS other all other tools is Log command. With help of this command user always have project history and this is invaluable - to have nice history instead of dozens of files with strange names. – Eleonoraeleonore

I understand. In fact a DVCS would actually be much more powerful, that's true. But there are also problems such as overhead, handling of big files (it wasn't made for that), cleaning the history as said suggested (again it wasn't made for that). The problem is that file systems are not powerful enough for you, but a DVCS would be a bit too much powerful (+requires work): what you need is something in between, some tool that can easily made local copies of files, with logs, and allows to choose and delete them at one point.. Maybe a new tool using git low-level fs-like features? Don't know.. – Anthony

Thank you, Christophe. Some new tool, that's what I'm trying to find here. But with no luck yet. May be git-bigfiles will solve the problems. Should check it. – Eleonoraeleonore

I have seen also gibak: eigenclass.org/hiki/gibak-backup-system-introduction or this fork: github.com/pangloss/gibak Good luck with your investigations and please post here whenever you find something suitable! :-) – Anthony

Why DVCS? .max files may be binary but you should be able to produce a good table of detailed metadata about the contents in the commit. A really advanced set up could even log the changes since the last commit. it won't help you with final merge but it would let you compare two metadata tables. Plus its better than saving a hundred max files in folder or subfolder. – Iverson

Answer 3

You may want to investigate Veracity which was designed with this requirement in mind. The Windows installer includes a Tortoise-like client. (I don't know if the Mac installer has a similar interface. I do know that the Linux installer currently doesn't have such an interface.)

Relevant links:

• Download
• Getting started (Windows client)
• Getting started (command line)
• DVCS comparison chart (note, chart created by the Veracity team) ... biggest differences from git are that you can't rebase - by design, no rewriting of history is allowed - and you can't bisect - don't know if this is by design or not

Prisoner

Answer 4

Have you considered using Autodesk Vault? It works well with 3dsmax (we've been using it for many years, both for Inventor and 3dsmax). http://www.autodesk.com/products/autodesk-vault-family/features

Perloff

Is vault still free? I think you have to buy it now. Plus I found it insulting to use unless you are in construction. – Iverson

And Vault is centralized like perforce (last time I checked) – Iverson

You can replicate server data between vault servers, we do this between servers located at different locations around the world, but the free version does not support this out of the box. – Perloff

The main reason I would want DVCS is so I can get the benefits of version control for myself but only push my commit when it's ready to share with the rest of the team. – Iverson

Answer 5

Regarding git destroying your binary max files,
did you set core.autocrlf=false?

GIT would be a pretty useless source control if it destroyed sources.
This question on our very own SO goes deeper into handling differences regarding CR & LF in GIT.

Birgit

No, I didn't. The failure happened two or three times, every time on shared folder and I couldn't reproduce it. Our company's staff is mainly artists so I should be totally confident in a software to encourage people to use it. If this kind of failure happens with someone's file it would be total disaster. To be true - I'm afraid of git a little. That's why I'm searching for alternate. – Eleonoraeleonore

I'll leave the answer because your comment could be of value for others. For all else, VonC is way better placed to help you with what you need (or don't need). – Birgit

Answer 6

TACTICS is also a backup system for graphics related projects, and its open source. It's worth a mention.

http://community.southpawtech.com/get-started

http://community.southpawtech.com/tactic-content/quickstart

Perloff