Existing solution for file deltas/versioning in Java

Question

When versioning or optimizing file backups one idea is to use only the delta or data that has been modified.

This sounds like a simple idea at first but actually determining where unmodified data ends and new data starts comes accross as a difficult task.

Is there an existing framework that already does something like this or an efficient file comparison algorithm?

Hauck

Answer 1

XDelta is not Java but is worth looking at anyway. There is Java version of it but I don't know how stable is it.

Rimrock

javaxdelta works fine, we are using it (with this wrapper) in production for directory diffs. xdelta is more advanced, but it's native and GPL – Goahead

Answer 2

Instead of rolling your own, you might consider leveraging an open source version control system (eg, Subversion). You get a lot more than just a delta versioning algorithm that way.

Wendling

Source Forge is a site which uses version control. For a version control system you can use subversion, CVS, git, mercurial etc. – Rosenstein

@Peter: I should never answer questions after midnight, thanks, I meant SVN. – Wendling

SVN has many uses, not just for development. I use it in production to deploy and version our configuration files. – Rosenstein

Answer 3

It sounds like you are describing a difference based storage scheme. Most source code control systems use such systems to minimize their storage requirements. The *nix "diff" command is capable of generating the data you would need to implement it on your own.

Lithosphere

Exact. This is to minimize the storage needed as keeping separate versions of the same file could eat up space quite fast. It would also be useful for backup schemes. – Hauck

Answer 4

Here's a Java library that can compute diffs between two plain text files:

http://code.google.com/p/google-diff-match-patch/

I don't know any library for binary diffs though. Try googling for 'java binary diff' ;-)

Numismatics

Binary diff is the search term I was looking for. Thanks for the tip. Will post back if I find a suitable framework. – Hauck

Answer 5

As for my opinion, Bsdiff tool is the best choice for binary files. It uses suffix sorting (Larsson and Sadakane's qsufsort) and takes advantage of how executable files change. Bsdiff was written in C++ by Colin Percival. Diff files created by Bsdiff are generally smaller than the files created by Xdelta.

It is also worth noting that Bsdiff uses bzip2 compression algorithm. Binary patches created by Bsdiff sometimes can be further compressed using other compression algorithms (like the WinRAR archiver's one).

Here is the site where you can find Bsdiff documentation and download Bsdiff for free: http://www.daemonology.net/bsdiff/

Sixgun