Can Git treat ZIP files as directories and files inside the ZIP as blobs?

Asked 3/11, 2011 at 20:48 Answered 24/3, 2023 at 9:40

The scenario

Imagine I am forced to work with some of my files always stored inside .zip files. Some of the files inside the ZIP file are small text files and change often, while others are larger but luckily rather static (e.g. images).

If I want to place these ZIP files inside a Git repository, each ZIP is treated as a blob, so whenever I commit the repository grows by the size of the ZIP file... even if only one small text file inside changed!

Why this is realistic

Microsoft Word 2007/2010 .docx and Excel .xlsx files are ZIP files...

What I want

Is there, by any chance, a way to tell Git to not treat ZIP files as files, but rather as directories and treat their contents as files?

The advantages

much smaller repository size, i.e. quicker transfer/backup
Display changes with Git to ZIP files would automagically work

But it couldn't work, you say?

I realize that without extra metadata this would lead to some amount of ambiguity: on a git checkout Git would have to decide whether to create foo.zip/bar.txt as a file in a regular directory or a ZIP file. However, this could be solved through configuration options, I would think.

Two ideas how it could be done (if it doesn't exist yet)

using a library such as minizip or IO::Compress::Zip inside Git
somehow adding a filesystem layer such that Git actually sees ZIP files as directories to start with

Indre answered 3/11, 2011 at 20:48 Comment(5)

The scenario with .docx files makes sense, but in many other cases you might want to consider tracking the individual files normally with git and only building the resulting .zip using an appropriate build tool like make. – Tamekia 28/11, 2013 at 10:11

Considering that two zip files that look different to each other can hold the exact same data (for example a text file zipped two times with two different compression levels), this becomes much trickier. While it is easy to represent the diff between the two versions of the unzipped files with little information, I guess representing the diff between the two versions of the archive (which is essentially what git has to do) with about as little information would be non-trivial. – Openhearted 23/12, 2013 at 14:53

Did you ever end up with an implemented solution of Jeff's answer or any thing else? I'm wondering about basically the same except for tar archives, which should yield a compatible answer... – Hemlock 3/5, 2016 at 9:56

SAP's Information Design Tool (IDT) creates a similar file structure for its UNX format. It's also recursive: it contains a BLX file and a DFX file, which are both archives, which correspond to is 'business layer' and 'data foundation', respectively. I'd like to have a solution as well. – Stonechat 2/3, 2017 at 22:22

Jetbrains build-in VCS does allow you to look inside zip type files. Very useful, but requires you to review e.g. PRs inside the IDE. Now that Microsoft has taken over, we might see this in the github pr diff as well. – Kreiner 21/9, 2018 at 12:57

This doesn't exist, but it could easily exist in the current framework. Just as Git acts differently with displaying binary or ASCII files when performing a diff, it could be told to offer special treatment to certain file types through the configuration interface.

If you don't want to change the code base (although this is kind of a cool idea you've got), you could also script it for yourself by using pre-commit and post-checkout hooks to unzip and store the files, then return them to their .zip state on checkout. You would have to restrict actions to only those files blobs / indexes that are specified by git add.

Either way is a bit of work -- it's just a question of whether the other Git commands are aware of what's going on and play nicely.

Surly answered 3/11, 2011 at 21:10 Comment(5)

Hooks do seem like a good direction to look in; I thought briefly of that but wasn't sure whether it could work. The pre-commit hook can modify both the file system and the staging area? – Indre 3/11, 2011 at 22:50

@Jonas Did you ever end up doing this and is there a chance of you posting a worked solution? I would love to usefully track changes to spreadsheets in git and CSV is just not fit for our purposes. – Lindeman 13/11, 2013 at 10:46

Note that using scripts that would unzip archived files before committing them to the repository and compress the files again upon checkout, a commit immediately followed by a checkout would be likely to modify the archives, even though the files stored inside of the archive would be unchanged. – Openhearted 23/12, 2013 at 15:6

I just wrote some hooks to do this. Still working on the rough edges, but could be helpful: github.com/ckrf/xlsx-git – Ether 24/4, 2015 at 5:39

I guess one of the advantages of zip files in this context is saving space on the remote repo. That's why I used them, at least. Your hooks won't seem to solve the problem, since the files would be stored and committed in their unzipped format. Am I right? I mean the ideal solution is to have git to delta zip files correctly, i.e., to delta its content and not the binary zipped file. – Mitra 28/10, 2022 at 14:12

Use bup (presented in details in GitMinutes #24)

It is the only git-like system designed to deal with large (even very very large) files, which means every version of a zip file will only increase the repo from its delta (instead of a full additional copy)

The result is an actual git repo, that a regular Git command can read.

I detail how bup differs from Git in "git with large files".

Any other workaround (like git-annex) isn't entirely satisfactory, as detailed in "git-annex with large files".

Oleic answered 21/11, 2013 at 18:57 Comment(7)

This seems very much geared towards very large files, the scenario was geared more towards XML such as docx and xlsx (which are frequently fairly small) zipped up. You'd get a smaller repo size with bup, but would you get to diff actual changes in the XML? – Lindeman 21/11, 2013 at 19:33

@Lindeman this is geared toward large files in size or in number. But it isn't much different from git in term of diff. – Oleic 21/11, 2013 at 19:56

Looks interesting, but can you use it with your actual git repo? – Surreptitious 20/3, 2015 at 14:20

@Surreptitious I don't think so: a bup repo is a git repo (raw.githubusercontent.com/bup/bup/master/DESIGN), but the reverse doesn't seem to be true. – Oleic 20/3, 2015 at 14:24

nice, it seems is already available in Ubuntu APT repo. But can you provide a TLDR on how to use it? The readme is lengthy and vague :) – Mitra 28/10, 2022 at 14:19

@JoãoPimentelFerreira 9 years later, I don't remember much about it. I have only the (lengthy) DESIGN page to go by. – Oleic 28/10, 2022 at 14:31

uau, didn't realize it was 9 years ago :) thanks anyway – Mitra 28/10, 2022 at 22:20

Zippey - A solution using Git file filter

My solution is to use a filter to "flatten" the ZIP file into an monolithic, expanded (may be huge) text file. During git add/commit the ZIP file will be automatically expanded to this text format for normal text diffing, and during checkout, it is automatically zipped up again.

The text file is composed of records, each representing a file in the ZIP file. So you can think this text file is a text-based image for the original ZIP file. If the file in the ZIP file is text indeed, it is copied into the text file; otherwise, it is Base64 encoded before copied into the text format file. This keeps the text file always a text file.

Although this filter does not make each file in the ZIP file a blob, text files are mapped line to line - which is the unit of the diff - while binary files changes can be represented by updates of their corresponding Base64. I think this is equivalent to what the OP imagines.

For details and a prototyping code, you can read the following link:

Zippey Git file filter

Also, credit to the place that inspired me about this solution: Description of how file filter works

Daggett answered 18/4, 2014 at 9:19 Comment(7)

This filter is still under development, if you have questions or any suggestions let me know. – Daggett 18/4, 2014 at 9:24

I tried this out and I think it should work well for me. I would just add something to the documentation that the text file list zippey.py has to be modified to include whatever file types you want zippey.py to recognize as text files. – Disapproval 22/11, 2014 at 1:33

Huge files like that are not friendly with many tools. I'm especially thinking of github 50 MB limit – Strepphon 19/4, 2018 at 10:10

I'm no fan of a monolithic file, because it would build a file too large to push to github (100MB), and does not allow fine tracking – Strepphon 26/6, 2018 at 14:7

some small improvements to zippey on my repo – Spiniferous 28/2, 2019 at 11:32

Here is a newer version: github.com/rockstorm101/zippey – Weise 1/11, 2022 at 15:48

From Managing ZIP-based file formats in git:

Note: per comment from Ruben, this is only about getting a proper diff though, not about committing unzipped files.

Open your ~/.gitconfig file (create if not existing already) and add the following stanza:

[diff "zip"]
textconv = unzip -c -a

What it does is using “unzip -c -a FILENAME” to convert your zipfile into ASCII text (unzip -c unzips to STDOUT). Next thing is to create/modify the file REPOSITORY/.gitattributes and add the following

*.pptx diff=zip

which tells git to use the zip-diffing description from the config for files matching the given mask (in this case everything ending with .pptx). Now git diff automatically unzips the files and diffs the ASCII output which is a little better than just “binary files differ”. On the other hand to to the convoluted mess that the corresponding XML of pptx files is, it doesn’t help a lot but for ZIP-files including text (like for example source code archives) this is actually quite handy.

Trilingual answered 28/11, 2013 at 11:8 Comment(3)

This is only about getting a proper diff though, not about committing unzipped files.. – Lindeman 28/11, 2013 at 12:44

Thanks. This answers the question I wanted to solve, of showing changes to the text files inside my gzip files when git diffing. I used [diff "gzip"] = zcat and *.gz diff=gzip. – Acuff 30/1, 2017 at 22:42

@Lindeman but does this commit only the delta into the git history? Or does this only help with git diff and it commits again the whole zip file even if only a small change is made? – Mitra 28/10, 2022 at 14:41

The java tool ReZipDoc, similar to Zippey by sippey, allows to handle ZIP files in a nicer way with Git.

How it works

When adding/committing a ZIP based file, Rezip unpacks it and repacks it without compression, before adding it to the index/commit. In an uncompressed ZIP file, the archived files appear as-is in its content (together with some binary meta information before each file). If those archived files are plain-text files, this method will play nicely with Git.

Benefits

The main benefit of Rezip over Zippey, is that the actual file stored in the repository is still a ZIP file. Thus, in many cases, it will still work as-is with the respective application (for example Open Office), even if it is obtained without going through a re-packing-with-compression filter.

How to use

Install the filter(s) on your system:

mkdir -p ~/bin
cd ~/bin

# Download the filer executable
wget https://github.com/costerwi/rezip/blob/master/Rezip.class

# Install the add/commit filter
git config --global --replace-all filter.rezip.clean "java -cp ~/bin Rezip --store"

# (optionally) Install the checkout filter
    git config --global --add filter.rezip.smudge "java -cp ~/bin Rezip"

Use the filter in your repository, by adding lines like these to your <repo-root>/.gitattributes file:

[attr]textual     diff merge text
[attr]rezip       filter=rezip textual

# Microsoft Office
*.docx  rezip
*.xlsx  rezip
*.pptx  rezip
# OpenOffice
*.odt   rezip
*.ods   rezip
*.odp   rezip
# Misc
*.mcdx  rezip
*.slx   rezip

The textual part is so that these files are actually shown as text files in diffs.

Spiniferous answered 28/2, 2019 at 11:58 Comment(1)

This sounds really cool! I haven't had the need for this in a while, so never got around to implementing something, but this would definitely be something I would try out. – Indre 6/1, 2020 at 9:37

Here is my approach:

Using Git diff filters for replacing the archive files with a content summary

git config filter.zip.clean "unzip -v %f | tail -n +4 | head -n -2 | awk '{ print \$7,\$8 }' | grep -vE /$ | LC_ALL=C sort -sfk 2,2"
git config filter.zip.smudge "cat"
git config filter.zip.required true

Using a pre-commit hook to extract and add the archive content:

#!/bin/sh
#
# Git archive extraction pre commit hook
#
# Created: 2021 by Vivien Richter <[email protected]>
# License: CC-BY-4.0
# Version: 1.0.2

# Configuration
ARCHIVE_EXTENSIONS=$(cat .gitattributes | grep "zip" | tr -d [][:upper:] | cut -d " " -f1 | cut -d. -f2 | head -c -1 | tr "\n" "|")

# Processing
for STAGED_FILE in $(git diff --name-only --cached | grep -iE "\.($ARCHIVE_EXTENSIONS)$")
do
    # Deletes the old archive content
    rm -rf ".$(basename $STAGED_FILE).content"
    # Extracts the archive content, if the archive itself is not removed
    if [ -f "$STAGED_FILE" ]; then
        unzip -o $STAGED_FILE -d "$(dirname $STAGED_FILE)/.$(basename $STAGED_FILE).content"
    fi
    # Adds extracted or deleted archive content to the stage
    git add "$(dirname $STAGED_FILE)/.$(basename $STAGED_FILE).content"
done

Using a post-checkout hook for packing the archives again for usage:

#!/bin/sh
#
# Git archive packing post checkout hook
#
# Created: 2021 by Vivien Richter <[email protected]>
# License: CC-BY-4.0
# Version: 1.0.0

# Configuration
ARCHIVE_EXTENSIONS=$(cat .gitattributes | grep "zip" | tr -d [][:upper:] | cut -d " " -f1 | cut -d. -f2 | head -c -1 | tr "\n" "|")

# Processing
for EXTRACTED_ARCHIVE in $(git ls-tree -dr --full-tree --name-only HEAD | grep -iE "\.($ARCHIVE_EXTENSIONS)\.content$")
do
    # Gets filename
    FILENAME=$(dirname $EXTRACTED_ARCHIVE)/$(basename $EXTRACTED_ARCHIVE | cut -d. -f2- | awk -F '.content' '{ print $1 }')
    # Removes the dummy archive file
    rm $FILENAME
    # Jumps into the extracted archive
    cd $EXTRACTED_ARCHIVE
    # Creates the real archive file
    zip -r9 ../"$FILENAME" $(find . -type f)
    # Jumps back
    cd ..
done

Apply the filter at the .gitattributes file:

# Macro for all file types that should be treated as ZIP archives.
[attr]zip text filter=zip

# Forces `LF` as line endings for text based files inside ZIP archives.
**/*.content/** text=auto eol=lf

# OpenDocument
*.[oO][dD][tT] zip
*.[oO][dD][sS] zip
*.[oO][dD][gG] zip
*.[oO][dD][pP] zip
*.[oO][dD][mM] zip

# Krita
*.[kK][rR][aA] zip

# VRoid Studio
*.[vV][rR][oO][iI][dD] zip
*.[fF][vV][pP] zip

Add some binary treatment to the .gitattributes file:

# Macro for all binary files that should use Git LFS.
[attr]bin -text filter=lfs diff=lfs merge=lfs lockable

# Images
*.[jJ][pP][gG] bin
*.[jJ][pP][eE][gG] bin
*.[pP][nN][gG] bin
*.[aA][pP][nN][gG] bin
*.[gG][iI][fF] bin
*.[bB][mM][pP] bin
*.[tT][gG][aA] bin
*.[tT][iI][fF] bin
*.[tT][iI][fF][fF] bin
*.[sS][vV][gG][zZ] bin

Add some stuff to the .gitignore file:

# Auto generated LFS hooks
.githooks/pre-push

# Temporary files
*~

Some configuration by:
1. Install Git LFS
2. Prepare LFS by issuing the command git lfs install once.
3. Setup the Git filter.
4. Install the hooks by issuing the command git config core.hooksPath .githooks.
5. Apply the checkout hook once by issuing the command .githooks/post-checkout.
6. Apply the filter once by issuing the command git add -A.

For an example see here: ZIP treatment for Git

Known issues

Whitespaces in filenames (22. April 2021), see: https://github.com/vivi90/git-zip/issues/1
Stash not working (30. October 2021), see: https://github.com/vivi90/git-zip/issues/4

Immigrate answered 8/3, 2021 at 13:0 Comment(1)

Have those issues been fixed? If so, can you update with version and date information? – Vista 6/11, 2021 at 21:1

Often there are problems with pre-zipped files for applications as they expect the ZIP compression method and file order to be the one they chose. I believe that OpenOffice .odf files have that problem.

That said, if you are simply using any old ZIP file as a method for keeping stuff together that you should be able to create a few simple aliases which will unzip and re-zip when required. The very latest MSysGit (aka Git for Windows) now has both zip and unzip on the shell code side, so you can use them in aliases.

The project I'm currently working on uses ZIP files as the main local version control / archive, so I'm also trying to get a workable set of aliases for sucking these hundreds of ZIP files into Git (and getting them out again ;-) so that the coworkers are happy.

Mediatize answered 3/11, 2011 at 21:36 Comment(1)

I just did a few tests for Word 2010 - it seems quite tolerant (deflate with different word sizes, deflate64 and changing file order in the zip file produced by 7zip all did not throw Word off). About using aliases, I was hoping to avoid any extra manual step... currently most of my commits go through TortoiseGit. – Indre 3/11, 2011 at 22:43

There is @callegar's Rezip implementation in Bash and I wanted to experiment to see if including the tool in the source repository (along with some script to configure git) would work.

I added the filter to the repository's config:

git config --replace-all filter.rezip.clean 'bash "$(git rev-parse --show-toplevel)/tools/rezip.sh" -p ODF_UNCOMPRESS2'
git config --replace-all filter.rezip.smudge 'bash "$(git rev-parse --show-toplevel)/tools/rezip.sh" -p ODF_COMPRESS2'

And noticed that zip utility is not available for plain Git-for-Windows installation even though unzip is. So I wrote a substitute in perl:

# zip() {
#   perl -e '
    use v5.35;
    use IO::Compress::Zip q/:all/;
    use File::Find;
    use Date::Parse qw/str2time/;

    {
      # needed a monkey-patch to clear file mtimes
      no warnings "redefine";
      my $zip_epoch = str2time("1980-01-01T00:00:00");
      my $t = IO::Compress::Zip::_unixToDosTime($zip_epoch);
      *IO::Compress::Zip::_unixToDosTime = sub { $t };
    }

    my $method = scalar(grep { /-0/ } @ARGV) ? ZIP_CM_STORE : ZIP_CM_DEFLATE;
    my $out = $ARGV[$#ARGV - 1];
    my $in = $ARGV[$#ARGV];

    my @files;
    find({ no_chdir => 1, wanted => sub { push @files, $_ if -f } }, $in);
    @files = sort @files;

    zip \@files, $out, Method => $method, CanonicalName => 1, Efs => 1, Minimal => 1;
#  ' -- $*;
# }

It's very slow, but kind of working. I think someone could reimplement the whole rezip in perl based on this.

Refined answered 24/3, 2023 at 9:40 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++