How to `git submodule add` Existing sub Repository?

Question

The Question

How to add existing sub repository as a submodule in git?

The Why

I have a private codespace supermodule with submodules scattered randomly:

codespace (git repo, private)
├── Archived_projects (git repos)
└── Projects
    ├── project-foo (git repo)
    └── project-bar (git repo)

Sometimes submodules have commits not ready to be pushed. But I want them to be saved while pushing supermodule codespace.
codespace is a repo cloned to c9.io workspace or other places.

What I Do

linus@machine /cygdrive/f/__Storage__/Workspace
$ git clone https://github.com/octocat/Spoon-Knife.git
Cloning into 'Spoon-Knife'...
$ cd Spoon-Knife/
$ git clone https://github.com/octocat/Spoon-Knife.git ./foo/bar
Cloning into './foo/bar'...
$ git add .

From cmd.exe

> git submodule add https://github.com/octocat/Spoon-Knife.git ./foo/bar
'foo/bar' already exists in the index
> cat .gitmodules
cat: .gitmodules: No such file or directory

From cygwin.exe (bash)

$ git submodule add https://github.com/octocat/Spoon-Knife.git ./foo/bar
': not a valid identifier/Git/mingw64/bin/gettext.sh: line 89: export: `sm_path
'' already exists in the index
$ cat .gitmodules
cat: .gitmodules: No such file or directory

Reference

git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>]
              [--reference <repository>] [--depth <depth>] [--] <repository> [<path>]

<repository> is the URL of the new submodule’s origin repository.

<path> is the relative location for the cloned submodule to exist in the superproject. If <path> does not exist, then the
submodule is created by cloning from the named URL. If <path> does exist and is already a valid Git repository, then this is
added to the changeset without cloning. This second form is provided to ease creating a new submodule from scratch, and
presumes the user will later push the submodule to the given URL.

In either case, the given URL is recorded into .gitmodules for use by subsequent users cloning the superproject. If the URL
is given relative to the superproject’s repository, the presumption is the superproject and submodule repositories will be
kept together in the same relative location, and only the superproject’s URL needs to be provided: git-submodule will
correctly locate the submodule using the relative URL in .gitmodules.

If <path> does exist and is already a valid Git repository, then this is added to the changeset without cloning.

Why this doesn't work in my case?

Ferric

This might help. Move the repo out of your worktree, submodule add that, fix up the urls – Mercorr

Answer 1

What went wrong

Where you went wrong is doing the

$ git add .

That adds everything, so also foo/bar, to the index of the current repository (ready to be committed thus).

Correct way

If you just don't do that and continue with

$ git submodule add https://github.com/CarloWood/XYZ.git foo/bar

then that should work; this would detect that foo/bar is an already cloned repository and add it to the current repository as a submodule.

Note that it is not needed to clone first. You explicitly say you already have done that, but for clarity for other readers I'd like to point out that if you omit the clone right before the git add . too (so there isn't a foo/bar at all now) then the above git submodule add ... would see there isn't anything yet and then simply clone it for you.

Note that there is a minor difference between methods. If you start with cloning then foo/.git will be a directory, while if you use git submodule add to do the cloning then this .git repository is put in .git/modules/foo of the parent project and foo/.git is a file containing the path to that. There is no real difference however as using a file for the .git to point anywhere else is generic and could be used anywhere; you can not conclude anything from .git being a file or directory.

Undressed

Thanks. I added it 5 years after it was asked - so it's not that strange they aren't monitoring it anymore :D. Hell maybe @Ferric has deceased in the meantime! – Undressed

Answer 2

One way to do it is to manually create the .gitmodules file.

The format is below:

[submodule "path/to/submodule1"]
    path = path/to/submodule/1
    url = [email protected]:user/submodule1
[submodule "path/to/submodule2"]
    path = path/to/submodule/2
    url = [email protected]:user/submodule2

Note you'll have to run

git submodule sync

to apply the changes

Shlomo

Note, it's .gitmodules. – Frumenty

Unfortunately this doesn't seem to do anything. – Waverly

@Waverly you need to run git submodule sync for the changes to be applied – Shlomo

@Shlomo I haven't tried but it sounds like it might. I ended up using this solution: https://mcmap.net/q/13388/-how-to-change-the-remote-repository-for-a-git-submodule ...which seems safer. – Waverly

Answer 3

git submodule add detects if the path given for a submosule exists and contains an initialized git repo, so no neeed to worry about that. I ran into a similar problem so I wrote a (hacky) script to deal with this issue.

#!/bin/bash
# save super directory
currentDir=`pwd`
# get all the git repos inside (except .) and format them 
# the way git does
gitDirs=`find -type d -name ".git" | sed -e 's|.git$||' -e 's|./||' -e 's|/$||' | grep -v "^$"`

for i in ${gitDirs[@]}
do
        echo "dealing with $i now"

        cd $i 
        # get the remote url for each submodule
        fetchUrl=`git remote -v | awk '/fetch/ {print $2}'`
        # for my purposes repos without remotes are useless
        # but you may have a different use case
        if [[ -z $fetchUrl ]]
        then
                echo "fetch url not found for this directory"
                continue
        else                                                      
                echo "got a fetch url of $fetchUrl for git repo $i"                                                                 
        fi                                                        

        cd $currentDir
        # make sure it isn't tracked as a submodule already                                                         
        existing=`grep -A5  $i ./.gitmodules | grep $fetchUrl`    

        if [[ -z $existing ]]                                     
        then                                                      
                echo "does not exist in .gitmodules yet, will create now"
                # if it doesn't exist yet then create it
                git submodule add $fetchUrl $i
        else
                echo "$i is already present as a submodule with fetch url: $fetchUrl"
                echo "The command we would have used is: git submodule add $fetchUrl $i"
        fi
done

Bettinabettine

Answer 4

Don't add the submodule using git add, use git submodule add

git submodule [--quiet] add [<options>] [--] <repository> [<path>]

If you've already cloned the repository, just run

git submodule add -- "$(git -C ./sub/ remote get-url origin)" ./sub/

NB: The relative path prefix, ./, is important here ... unless you are specifying an absolute path.

Alternatively

git submodule add -- ./sub.git ./sub

if the project is using relative URLs.

Liger

It is important to note that this will add a submodule that literally points to the local working copy, and not to its remote. You can see this for yourself by looking inside the updated .gitmodules file. – Anselmo

True, the local repo is added as the repo. To change it after the fact change such configuration using git submodule set-url -- sub "$(git -C ./sub/ remote get-url origin)". – Liger

Answer 5

To augment Carlos Answer, for an existing repo. It already contains its URL. You can simply

 git submodule add $(cat repodir/.git/config  | grep url | cut -d '=' -f 2) reponame

Implore

Maybe instead of parsing the config file yourself, you could use git config, to read the url: git submodule add $(git -C repodir config remote.$(git -C repodir config branch.master.remote).url) repodir. This selects the url of the remote tracking branch of the master branch. (Works even if multiple remotes exist, your example doesn't work in that case.) (git -C somedirectory means, that git behaves as if started from within somedirectory) – Gadson

unnecessarily complex, git -C sub/ remote get-url origin | xargs -I% git submodule add % ./sub/ – Liger

even easier, git submodule add ./sub/, NB ./ – Liger

Answer 6

1. remove files from index foo/bar: git rm -f --cached foo/bar
2. git submodule add https://github.com/octocat/Spoon-Knife.git ./foo/bar

Sandal

Answer 7

At least if the (future) submodules do not have recursive other repositories/.git directories then the following simply adds all of them. (I am not sure why I had to supply the directory as path and "repository" to the command but whatever...)

for d in $(dirname $(find -mindepth 2 -name .git)) ; do git submodule add -- ./$d/ ./$d/ ; done

Immoderacy

Answer 8

You don't need to clone manually first. After running git submodule add [url] [path] run git submodule update, which will clone/pull all submodules for you.

Incoordination

I have repository with cloned and modified data already. Ideally I would like to be able to push submodules with supermodule to the server. Well, if it is not possible, I accept your answer. – Ferric

I've got same question, already cloned, and modified some files, how to add this modified repo as submodule to another git repo? – Irmgard