git archive export with submodules (git archive all / recursive)

Question

I have a website directory versioned with git. I use submodules for required libraries like Twitter Bootstrap, colorbox and lessjs because I should not track the sourcecode but only the version of their code I use.

Now I want to deploy the project and therefore I need the submodule code too. By using git archive I don't get the source files/code from the submodules.

The following three approaches TRY to achieve what I want but dont't work: 1st approach

#!/bin/sh

export revision="$1"

export GIT_INDEX_FILE=".git/tmpindex"
rm -f "$GIT_INDEX_FILE"

git read-tree $revision

export up="$(pwd)"

read_one_level () {
        export GIT_ALTERNATE_OBJECT_DIRECTORIES="$GIT_ALTERNATE_OBJECT_DIRECTORIES":$(
            git submodule foreach 'echo "$up/$path/.git/objects"' |
            grep -E -v '^(Entering|No submodule mapping found)' |
            tr '\n' : |
            sed 's/:$//'
        )

        git submodule foreach '
                cd "$up"
                subcommit=$(git rev-parse :"$path")
                git rm --cached "$path"
                git read-tree -i --prefix="$path/" $subcommit
        ' >/dev/null
}

while git ls-files -s | grep -q ^160000; do
    read_one_level
done

git archive --format=tar $(git write-tree)

rm -f "$GIT_INDEX_FILE" 

Thomas Rast in http://git.661346.n2.nabble.com/Running-git-archive-recursively-over-submodules-td4577012.html

This gives me errors both on Windows as in Linux that there are no objects files found.

2nd approach https://github.com/meitar/git-archive-all.sh

Complains about mktemp not found on Windows. And correcting calls to git archive (from git-archive) does not give submodules in the tar ...:(

3rd approach https://github.com/Kentzo/git-archive-all

Is outdated from my point of view by not being compatible to latest python 3.3 and still not fully working by using 2.7 because samefile complains.

So my question is now: Is there any recent way/approach to deal with exporting/archive a git project including submodules?

Or should I check subtrees for this workflow?

Thanks in advance

Kilian

I'm the author of Kentzo/git-archive-all. It's python 3.3 compatible now. What is the other issue? – Cheslie

Answer 1

https://github.com/Kentzo/git-archive-all is now working under linux as @Kentzo stated.

In the cygwin environment (with git and python installed via cygwin installer) it works there too.

Kilian

git-archive-all is available at (pypi.org/project/git-archive-all), install it with pip install git-archive-all... but caution: takes as output file only .tgz format. – Pinnule

Sorry, I just detected that my system (Fedora 33) had installed version 1.17 ... the latest version has no problems with tar, tar.gz, tar.xz and so on. – Pinnule

Answer 2

I'm using the following code

git archive -o release.zip HEAD
git submodule --quiet foreach 'cd $toplevel; zip -ru release.zip $sm_path'

to create a complete archive of a git repository with all submodules.

If you want to be fancy you can even rewrite the zip comment via

echo -e "Repository:\n$(git rev-parse HEAD)\nSubmodule status:\n$(git submodule status)" | zip -u release.zip -z

All on windows using infozip.

Premier

$path is deprecated, the recommendation is to use $sm_path instead (to avoid clashes with the PATH variable in Windows. – Alonso

@BramSchoenmakers Thx, fixed. – Premier

This is not honoring .gitignore for the submodules. – Lurette

This also includes the .git file from the submodule directory. Good enough though. – Candidacandidacy

Answer 3

If you are a friend of KISS as me you could use the answer of @t-b BUT as I found this solution does not archive nested submodules. The following code will help

# archive main directory
$ git archive --format tar -o release.tar HEAD
# descend recursively and archive each submodule
$ git submodule --quiet foreach --recursive 'git archive --format tar --prefix=$displaypath/ -o submodule.tar HEAD'
# concatenate with main archive
$ TOPDIR=$(pwd) git submodule --quiet foreach --recursive 'cd $TOPDIR; tar --concatenate --file=release.tar $displaypath/submodule.tar; rm -fv $displaypath/submodule.tar'
$ gzip -9 release.tar

The result will be file result.tar.gz... instead of HEAD you could choose another commit, of course.

Pinnule

Answer 4

We could do a quick bash script, where archive in tar, module and sub-modules, after we concatenate tars and compress, here it is an example with 2 sub modules:

#!/bin/bash

set -e

pwd=$(pwd)
date=$(date +%Y%m%d)
package=gitreponame
branch=master
name=tarbz2name
submodule1=/src/app/sub1/
submodule2=/src/sub2/

pushd ${package}
git checkout ${branch}
git pull
tag=$(git rev-list HEAD -n 1 | cut -c 1-7)
git archive --prefix="${name}/" --format=tar master > "$pwd"/${name}-${date}-${tag}.tar

git submodule update --init

cd ${submodule1}
git archive  --prefix="${name}${submodule1}" --format=tar master > "$pwd"/sb1.tar
cd -

cd ${submodule2}
git archive --prefix="${name}${submodule2}" --format=tar master > "$pwd"/sb2.tar
popd

tar -Af ${name}-${date}-${tag}.tar sb1.tar
tar -Af ${name}-${date}-${tag}.tar sb2.tar
bzip2 ${name}-${date}-${tag}.tar
rm sb1.tar sb2.tar

Lynching

Answer 5

The 2nd approach: https://github.com/meitar/git-archive-all.sh deploys as a standalone Bash script and appears to be still-maintained. I found it to be an easy way to take a quick submodule inclusive snapshot of a project. It can even capture an earlier revision (sort of, see below):

git-archive-all.sh --tree-ish (tag) ## for example

A couple of problems that may or may not be deal breakers:

• I ran into problems trying to use the --format option -- but the default, tar, generally worked.

• The (very) big problem: It captured my submodule, but only in its current version -- not the specified version. Whoops!

Being a Bash script, this probably requires some kind of Linux compatibility layer (like Cygwin) to be run under Windows.

---

It seems like this should be a built-in capability of Git -- it's too complicated to piece together with scripting. I guess not many people use submodules or git-archive, so even fewer want to use both.

Ann

The noted benefit of this approach was it can even capture an earlier revision (although it turned out to not really work). I don't see how to do that automatically with the @Cheslie solution either . The @Sergio solution appears to (sort of) do this, but I haven't yet tested it myself. I guess you have to do an actual checkout (and submodule update) with all of these solutions -- something that isn't required for a regular git archive. – Ann

Answer 6

I have developed git-archive-with-submodules using bash. It also has the nice feature of allowing to create an archive including uncommitted changes.

Affright

Answer 7

A simple solution that recursively calls git archive to zip up submodules.

The script should be placed in the Git repository's root directory (i.e. where the .git file is located).

The script hard-codes the following:

• path to script
• path to export directory.

#!/bin/bash
###
# Script name: archive_git_repos_as_zips.sh
# Purpose: Recursively produce zip archives of nested Git repositories.
###

# Path to this script
export SCRIPT='/PATH/TO/SCRIPT/archive_git_repos_as_zips.sh'

# Target directory for storing zip files
ZIP_DIR='/PATH/TO/git_zip_exports'

PWD=`pwd`
THIS=`basename $PWD`

# Zip file path
ZIP_FILE="$ZIP_DIR/${THIS}.zip

echo -e "\nCreating: $ZIP_FILE"
git archive -o $ZIP_FILE HEAD
git submodule --quiet foreach '$SCRIPT'

# end #

Bulbar