Restore git submodules from .gitmodules

Question

I have a folder, which was a git repo. It contains some files and .gitmodules file. Now, when I do git init and then git submodule init, the latter command output is nothing. How can I help git to see submodules, defined in .gitmodules file without running git submodule add by hand again?

Update: this is my .gitmodules file:

[submodule "vim-pathogen"]
    path = vim-pathogen
    url = git://github.com/tpope/vim-pathogen.git
[submodule "bundle/python-mode"]
    path = bundle/python-mode
    url = git://github.com/klen/python-mode.git
[submodule "bundle/vim-fugitive"]
    path = bundle/vim-fugitive
    url = git://github.com/tpope/vim-fugitive.git
[submodule "bundle/ctrlp.vim"]
    path = bundle/ctrlp.vim
    url = git://github.com/kien/ctrlp.vim.git
[submodule "bundle/vim-tomorrow-theme"]
    path = bundle/vim-tomorrow-theme
    url = git://github.com/chriskempson/vim-tomorrow-theme.git

and here is listing of this dir:

drwxr-xr-x  4 evgeniuz 100 4096 июня  29 12:06 .
drwx------ 60 evgeniuz 100 4096 июня  29 11:43 ..
drwxr-xr-x  2 evgeniuz 100 4096 июня  29 10:03 autoload
drwxr-xr-x  7 evgeniuz 100 4096 июня  29 12:13 .git
-rw-r--r--  1 evgeniuz 100  542 июня  29 11:45 .gitmodules
-rw-r--r--  1 evgeniuz 100  243 июня  29 11:18 .vimrc

so, definitely, it is in top level. the git directory is not changed, only git init is done

Parisparish

Are the submodules already present, in the sense that if you change into any submodule directory there are files present, and git rev-parse --show-toplevel gives you the submodule rather than "supermodule" directory? – Homiletics

no, submodule dirs are not present. imagine this folder is completely empty with just .gitmodules file – Parisparish

Ah, I see what the problem is - I've updated my answer. – Homiletics

Answer 1

git submodule init only considers submodules that already are in the index (i.e. "staged") for initialization. I would write a short script that parses .gitmodules, and for each url and path pair runs:

git submodule add <url> <path>

For example, you could use the following script:

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key local_path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $local_path
    done

This is based on how the git-submodule.sh script itself parses the .gitmodules file.

Homiletics

do you mean git submodule update --init? all of these commands fail silently. It looks like git doesn't see modules defined only in .gitmodules – Parisparish

After the git submodule init, do you see any output from git config --list | grep submodule? (As the documentation says, git submodule init should "Initialize the submodules, i.e. register each submodule name and url found in .gitmodules into .git/config".) – Homiletics

no, nothing. git config --list gives only standard values, no mention of submodules – Parisparish

@Shark: that's strange - could you update your question with the contents of your .gitmodules file? Also, is your .gitmodules file definitely at the top level of your repository, rather than a subdirectory? – Homiletics

@Shark: yeah, me too - and I may well be missing something obvious! However, I think the idea of initializing from .gitmodules is that it will have been committed in the same commit (or close to the same commit) as one that added the submodules to the tree, so having only .gitmodules, but no submodules in the index, is an unusual situation to be in. – Homiletics

git submodule update --init -f - will fork submodules to its dirs permomently. – Asphyxiant

One of the problems here is that because the original .git is not available, you don't know which commit-id the submodule was pinned to, this makes it difficult to acquire the exact revision of the submodule that you had in the original repository. – Loaning

Has anyone got a solution for parsing the branch in the .gitmodules file and adding the submodule using the correct branch? – Scandinavia

Unfortunately, if the submodule contains spaces like after git submodule add URL 'path with spaces' this solution fails. Perhaps try something like git config -f .gitmodules -z --get-regexp '^submodule\..*\.path$' | sed -z 's/\n.*$//' | tr '\0' '\n' | while read keys; .. – Thermionic

If some submodules are already in the index, the script will exit. To avoid that: git submodule add $url $path || true – Antoneantonella

Answer 2

Expanding on @Mark Longair's answer, I wrote a bash script to automate steps 2 & 3 of the following process:

1. Clone a 'boilerplate' repo to begin a new project
2. Remove the .git folder and re-initialize as a new repo
3. Re-initialize the submodules, prompting for input before deleting folders

---

#!/bin/bash

set -e
rm -rf .git
git init

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' > tempfile

while read -u 3 path_key path
do
    url_key=$(echo $path_key | sed 's/\.path/.url/')
    url=$(git config -f .gitmodules --get "$url_key")

    read -p "Are you sure you want to delete $path and re-initialize as a new submodule? " yn
    case $yn in
        [Yy]* ) rm -rf $path; git submodule add $url $path; echo "$path has been initialized";;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac

done 3<tempfile

rm tempfile

Note: the submodules will be checked out at the tip of their master branch instead of the same commit as the boilerplate repo, so you'll need to do that manually.

Piping the output from git config into the read loop was causing problems with the prompt for input, so it outputs it to a temp file instead. Any improvements on my first bash script would be very welcome :)

---

Big thanks to Mark, https://mcmap.net/q/12175/-how-do-i-prompt-for-yes-no-cancel-input-in-a-linux-shell-script, bash: nested interactive read within a loop that's also using read, and tnettenba @ chat.freenode.net for helping me arrive at this solution!

Huynh

Answer 3

Extending excellent @Mark Longair's answer to add submodule respecting branch and repo name.

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        name=$(echo $path_key | sed 's/\submodule\.\(.*\)\.path/\1/')
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        branch_key=$(echo $path_key | sed 's/\.path/.branch/')
        url=$(git config -f .gitmodules --get "$url_key")
        branch=$(git config -f .gitmodules --get "$branch_key" || echo "master")
        git submodule add -b $branch --name $name $url $path || continue
    done

Mclyman

This worked great, however it also duplicated each entry in the .gitmodules file which needs some manual cleanup after the fact. – Moorwort

Answer 4

An updated version of the script by @mark-longair . This one also supports branches, handles the case where some of the submodules already exist in .git/config, and when necessary backs up existing directories of the same name as the submodule paths.

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/');
        branch_key=$(echo $path_key | sed 's/\.path/.branch/');
        # If the url_key doesn't yet exist then backup up the existing
        # directory if necessary and add the submodule
        if [ ! $(git config --get "$url_key") ]; then
            if [ -d "$path" ] && [ ! $(git config --get "$url_key") ]; then
                mv "$path" "$path""_backup_""$(date +'%Y%m%d%H%M%S')";
            fi;
            url=$(git config -f .gitmodules --get "$url_key");
            # If a branch is specified then use that one, otherwise
            # default to master
            branch=$(git config -f .gitmodules --get "$branch_key");
            if [ ! "$branch" ]; then branch="master"; fi;
            git submodule add -f -b "$branch" "$url" "$path";
        fi;
    done;

# In case the submodule exists in .git/config but the url is out of date

git submodule sync;

# Now actually pull all the modules. I used to use this...
#
# git submodule update --init --remote --force --recursive
# ...but the submodules would check out in detached HEAD state and I 
# didn't like that, so now I do this...

git submodule foreach --recursive 'git checkout $(git config -f $toplevel/.gitmodules submodule.$name.branch || echo master)';

Inertia

Answer 5

I had a similar issue. git submodule init was failing silently.

When I did:

git submodule add <url> <path>

I got:

The following path is ignored by one of your .gitignore files: ...

I'm thinking that .gitignore(d) paths might be the cause.

Paperboard

Answer 6

I'm so surprised to see so many scripts for a very simple action and unfortunately I haven't found any tool or ready for use command. The only software so far which allows to restore it - is Gitkraken. Check here: https://help.gitkraken.com/gitkraken-client/submodules/

Cyma

Actually I have found one: npx @iclare/ksync .gitmodules . --nobackup – Cyma

Answer 7

I know its been a while, but I want to share this version that calls git config only once, doesn't requires a script and also handles branches:

git config -f .gitmodules --get-regexp '^submodule\.' | perl -lane'
$conf{$F[0]} = $F[1]}{
@mods = map {s,\.path$,,; $_} grep {/\.path$/} keys(%conf);
sub expand{$i = shift; map {$conf{$i . $_}} qw(.path .url .branch)}
for $i (@mods){
    ($path, $url, $branch) = expand($i);
    print(qq{rm -rf $path});
    print(qq{git submodule add -b $branch $url $path});
}
'

The only side effect is the output of the commands, nothing gets executed, so you can audit before committing to them.

This works with a simple copy and paste at the console, but should be trivial to put in a shell script.

example output:

rm -rf third-party/dht
git submodule add -b post-0.25-transmission https://github.com/transmission/dht third-party/dht
rm -rf third-party/libutp
git submodule add -b post-3.3-transmission https://github.com/transmission/libutp third-party/libutp
rm -rf third-party/libb64
git submodule add -b post-1.2.1-transmission https://github.com/transmission/libb64 third-party/libb64
rm -rf third-party/libnatpmp
git submodule add -b post-20151025-transmission https://github.com/transmission/libnatpmp third-party/libnatpmp
rm -rf third-party/miniupnpc
git submodule add -b post-2.0.20170509-transmission https://github.com/transmission/miniupnpc third-party/miniupnpc
rm -rf third-party/libevent
git submodule add -b post-2.0.22-transmission https://github.com/transmission/libevent third-party/libevent

Brennan

Answer 8

For zsh users, try my function which has DRY_RUN=1 support to see what commands will be ran and only uses git to parse the file instead of sed.

gsub_file() {(
  set -eu

  cd "$(git rev-parse --show-toplevel)"

  submodule_paths=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "path" | awk '{ print $2 }')}"
  )
  submodule_urls=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "url" | awk '{ print $2 }')}"
  )
  submodule_branches=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "branch" | awk '{ print $2 }')}"
  )

  sh_c() {
    echo + "$*"
    if [ "${DRY_RUN-}" ]; then
      return
    fi
    eval "$@"
  }

  for (( i=1; i <= ${#submodule_paths[@]}; i++ )) do
    p="${submodule_paths[$i]}"
    if [ -d "$p" ]; then
      continue
    fi

    url="${submodule_urls[$i]}"
    unset b
    if [ "${submodule_branches[$i]-}" ]; then
      b="-b ${submodule_branches[$i]}" 
    fi
    sh_c git submodule add "${b-}" "$url" "$p"
  done
)}

Glister