Automatically Add All Submodules to a Repo

Question

I have a big Directory ~/.vim and in it I have a subdirectory with many other git repos in it. I want to make a git repo of my ~/.vim directory though, but don't want to go through each of my other git subdirectories.

Is there any way of just recursively going through and adding all submodules?

Erato

You want make your .vim folder a git repo and add all your subdirectories (which are git repos) as submodules? Is that right ? – Carolyn

yeah, i've got ~/.vim/bundle/matchit for example which is cloned straight from github – Erato

Answer 1

Suppose that .vim is already a valid git repo and your want to add all git repos to your main git repo, then the for loop below is probably what you need:

First, cd to the root of your git repository.

Paste-able preview command- echo only, won't make any changes:

for x in $(find . -type d) ; do if [ -d "${x}/.git" ] ; then cd "${x}" ; origin="$(git config --get remote.origin.url)" ; cd - 1>/dev/null; echo git submodule add "${origin}" "${x}" ; fi ; done

Paste-able command to add the submodules:

for x in $(find . -type d) ; do if [ -d "${x}/.git" ] ; then cd "${x}" ; origin="$(git config --get remote.origin.url)" ; cd - 1>/dev/null; git submodule add "${origin}" "${x}" ; fi ; done

This loop first finds directories only, looks for a .git directory, identifies the original URL and then adds the submodule.

Readable version:

for x in $(find . -type d) ; do
    if [ -d "${x}/.git" ] ; then
        cd "${x}"
        origin="$(git config --get remote.origin.url)"
        cd - 1>/dev/null
        git submodule add "${origin}" "${x}"
    fi
done

Carolyn

The following is what worked for me: for i in $(ls -d bundle/*); do;if [ -d "$i"/.git ]; then; git submodule add $(cd $i && git remote show origin | grep Fetch | awk '{print $3}') ./$i; fi; done – Hasan

Change the initial for loop's find command to for x in $(find . -mindepth 1 -maxdepth 1 -type d), the -mindepth is needed to skip the current top level directory (the '.'), while maxdepth doesn't allow find to go deeper in dir structure. You might want to skip the -maxdepth if you have deeply nested repos. – Photopia

Answer 2

cd /to/super/repo
root_path=$(pwd)
for submodule in $(find -mindepth 2 -type d -name .git); do
    submodule_dir=$root_path/$submodule/..
    remote_name=$(cd $submodule_dir && git rev-parse --abbrev-ref --symbolic-full-name @{u}|cut -d'/' -f1 )
    remote_uri=$(cd $submodule_dir && git remote get-url $remote_name)

    echo "Adding $submodule with remote $remote_name..."

    # If already added to index, but not to .gitmodules...
    git rm --cached $submodule_dir &> /dev/null

    git submodule add $remote_uri $submodule_dir
done

Arturoartus

Answer 3

I had git repos nested in deeper sub-directories, so to find and add them from any level of depth, I had to use: for i in $(find ~/Sources/* -type d -name .git) do cd $i && cd .. && git submodule add $(pwd) done

Deferent

I doubt that this does something useful. This adds the submodules with a local path as remote, but with a http or ssh URI. – Arturoartus

Answer 4

sadly, this feature doesn't seem to exist. git should have a command that can add all the modules referenced in .gitmodules, something like:

### wishful thinking

git submodules init --all
git submodules add --all

:(

Scheers

Answer 5

Since doing this can be a repetitive task in many projects, I have created two helper functions based on awesome @Fatih answers with few modifications and store them in my machine using a custom bash shell file.

These functions can be invoked in any folder without the need of remembering or typing all of the implementation details.

Here are the steps to make life easier:

In your user root path create a file something like

.myBashCommands.sh

Write the following contents in it, If you already have your own dedicated file for custom functions, feel free to use it instead.

#!/bin/bash

# Find and log all git sub-modules if exist without making any changes.
# Credits: https://mcmap.net/q/1413790/-automatically-add-all-submodules-to-a-repo
function logAllGitSubModulesIfExist() {
 for x in $(find . -type d) ; do
 if [ -d "${x}/.git" ] ; then
     cd "${x}" ;
     origin="$(git config --get remote.origin.url)" ;
     cd - 1>/dev/null;
     echo "found gitsubmodule at =>" "${origin}" "${x}" "\n";
 fi ;
 done
}

# This loop first finds directories only, looks for a .git directory, identifies the original URL and then adds the submodule.
# Credits: https://mcmap.net/q/1413790/-automatically-add-all-submodules-to-a-repo
function addAllGitSubModules() {
for x in $(find . -type d) ; do
    if [ -d "${x}/.git" ] ; then
        cd "${x}"
        origin="$(git config --get remote.origin.url)"
        cd - 1>/dev/null
        git submodule add "${origin}" "${x}"
    fi
done
}

Close the file and while you are still in the same working directory, Refresh your terminal so that it recognize your newly added methods.

Here is how to do it: -

source .myBashCommands.sh

That's it! Now you are ready to reuse the two functions in any project and in any folder.

To invoke the methods simply type the name.

For example to log all available submodules just type the following in your terminal

logAllGitSubModulesIfExist

Similarly to find and add all existing submodules, just type the following: -

addAllGitSubModules

Coracoid

Answer 6

@Mussa When I am doing it it's showing me

The following path is ignored by one of your .gitignore files:

Use -f if you really want to add it.

Zonate

Answer 7

I would recommend to create a python script

import 
glob,remodules = glob.glob('*/**/.git', recursive=True)
content = ''def get_url(submod):
    for line in open(submod+'/config','r'):
       if '\turl' in line:
           return line.split('=')[1].strip()
for submod in modules:
    content += f'''
[submodule "{submod.strip(".git/")}"]
    path = {submod.strip(".git/")}
    url = {get_url(submod)}
    '''
print(content)
with open('.gitmodules','w') as f:
    f.write(content)

and push the changes into your github using the following command

git add . 
git commit -m "Adding submodule"
git push origin branch_name 

There might be chances that some of the files will be not added , you can simply add the generated .gitmodules file and add them manually. In case you are seeing that some of the files are giving error either edit the above code according to the location or you can delete them temporarily and add them manually to get best out of it.

example of .gitmodules

[submodule "src/abc"]
    path = src/abc
    url = https://github.com/abc.git

Zonate