How to add already cloned projects as submodules?

Question

I have in a folder a large number of projects that I cloned for quite some time; recently I moved this whole folder into one of my repos and would like to convert these cloned projects into submodules so that I can better update and control in the future. I've googled a lot on how to do this, but all the tutorials only talk about how to add submodules fresh. Anyone can help me?

Prefigure

Answer 1

Adding existing git repository as a submodule is the same as adding a new one.

1. First make the folder that contains all your git repositories as itself a git repository with git init.
2. Use git submodule add https//example.com/remoterepo ./localrepo, where ./localrepo is your existing git repository.

• Note: You get the URL for remoterepo from localrepo/.git/config.

3. Repeat the second step for all your existing repositories that you want to add as submodules.
4. Now you can run git submodule foreach git pull to update all your subprojects.

You may want to write a small script to automate the second step if you have lots of submodules, which shouldn't be difficult.

#Edit The following is what I used for trying to reproduce the error mentioned in the comments. I triple checked the commands, and I still can't see the error:

git --version

mkdir submoduletest
cd submoduletest

git init --bare remote_repo_A
git init --bare remote_repo_B 

git clone remote_repo_A local_repo_A
git clone remote_repo_B local_repo_B

cd local_repo_A
echo "test commit on repo B" >> test.txt
git add test.txt
git commit -m 'test commit message'
git push origin master

cd ../local_repo_B
echo "test commit on repo B" >> test.txt
git add test.txt
git commit -m 'test commit message'
git push origin master

cd ../local_repo_A
git clone ../remote_repo_B local_repo_B
git submodule add ../remote_repo_B ./local_repo_B
git submodule foreach git pull origin master

git add .
git ci -m 'we just added a submodule for remote_repo_B'

git submodule status

Use the following command to check current status of local_repo_A, it has only two blob objects, one for 'test.txt' and another for the implicitly created '.gitmodules' file, nothing from remote_repo_B are added to the index of local_repo_A.

find .git/objects -type f | awk -F/ '{print $3$4}' | xargs -I {} git cat-file -t {} | grep blob

Aglow

It does look like the correct solution. I was in a hurry last night and decided to remove & re-add all the repositories in the end. But still thank you for your answer! Although I wonder if there's anyway such that you don't need to enter the URL (the remote URL should be present in existing repo somewhere, just have no idea how to extract that..) – Prefigure

I get a "already exists in the index" error. @lynnard Did you test the answer before checking it? If not, then I urge you to do so because this is misleading – Raid

@lynnard I actually got around the problem, and suggested an edit. Cheers! – Raid

@MuhsinFatih, You get the "already exists in the index" error because you didn't follow the instructions as is, you must have added the sub repos as normal directories to the master repo before adding them as submodules, in that case they will not be treated as submodules. By the way, the edit is rejected because that is not part of the answer but something one should learn about the submodule command or git in general. – Aglow

@Aglow But that is not what the OP has asked for. He said he already has a git repository containing the other git repositories which will be difficult to download again. Besides since I said "already cloned repository" for step #2, if a person has none, they can simply skip the step. Additionally, your statement: "Adding existing git repository as a submodule is the same as adding a new one" is incorrect, and conflicts with your comment; Do you want to add "existing git repositories" in the repository as submodules, or do you want to add "new get repositories" as submodules? – Raid

I don't think the statement is incorrect, the commands were tested. And adding existing repos uses the exact same command as adding new ones(not cloned yet), this is tested too. As the OP asked, "moving a bunch of repos to an existing repo", since the destination is already a git repo, no git init is needed, simply git submodule add will do, no git add . is needed before executing the submodule command(which you most probably did in your test). – Aglow

@Aglow "simply git submodule add will do" no it will not. I just tested it, multiple times to make sure I am not mistaken. If you have a git repository B cloned in another git repository A, then trying to make B a submodule with your answer WILL generate an error. You rolled back two approved edits? Wow thats very nice. Fine, you may not accept the error in your answer, you may roll back approved edits that only add value to answer, not brake it in either case, then you should welcome when I copy your answer and add add my edit. – Raid

@MuhsinFatih, this is my last comment on this answer, you can play the commands and prove yourself right. – Aglow

@Aglow I am not trying to prove myself right. You should ask that question to yourself, as I am not the one who rolled back two approved edits. To be honest this confuses me because you have good reputation. In that case I assume you know the community rules. If an edit is good, it gets approved. What makes SO great is it's hard rules that encourage best answers to show up on top. I tested your answer, improved it, edited it, and you rolled it back. I did what I believed is the best for the community. If you have a problem with that then flag my answer rather than whining about it. – Raid

@Aglow write my name with big title and insult me. Great. Now that's a professional. If my answer is wrong, you could have simply downvote it. I flagged your answer. Thanks for your kind comments – Raid

Looks to me like the author here is trying to address your comments via an edit, which is a good way of doing so, @muhsin; I don't see any insults. You may want to walk away from this for a bit and revisit it later with fresh eyes. – Hooded

@Hooded Here I quote: "you never get it right even I explained, and that is your problem, I really don't want to waste time on this simple question, don't mess with my answer if you can't understand it." Also name-calling is described as inappropriate behavior under stackoverflow.com/help/be-nice, and before this humiliation he wrote my name as a big header. Imagine this being done to you, he's basically telling me that I am too dumb to understand his answer. I have no problem with the test case. Just remove what was said to me – Raid

@MuhsinFatih, I am sorry about that if that is insult to you, I will remove your name from the edit. What I said was based on test, and I provided the commands I used for the tests. Anyone tell me that the answer is incorrect, prove it, and if it is true, I will be very happy to accept and learn, otherwise, leave the answer alone. – Aglow

It's been a few years, I stumbled across this post again. @Aglow I just wanted to acknowledge that I was wrong to suggest the edits. Apologies for the trouble earlier. Reading this whole convo honestly embarrasses me today, I wish I can delete it all lol – Raid

@Raid Never mind :-) – Aglow

Answer 2

To add many submodules, I wrote this simple loop:

for repo in vim/bundle/*
do
  echo $repo
  pushd $repo
  url=$(git remote get-url $(git remote))
  echo $url
  popd
  git submodule add $url ./$repo
done

Obvious limitations I didn't bother fixing:

• don't need to change directory, pretty sure you can pass the git directory as an argument to commands
• git remote actually returns all remotes, not just the current, so script will break if there are multiple

Thyratron

As for the first limitation: pushd and popd should be replacable by the -C <path> option of git, i.e., git -C $repo submodule add $url ./$repo. The second limitation should be solvable using for remote in $(git remote); do ...; done where ... is the url variable declaration, echo and submodule add. But as I didn't test it (yet), I won't edit the answer for this. – Hairball

Answer 3

I will add onto @neevek's answer, if you get the already exists in the index error, then you can try this version:

1. Remove every "already cloned repository" from cache:
   (Because there is already an object there that does not point to the repository as a submodule, and it has to be deleted from index first to prevent conflict. Your repositories' contents or index will not be deleted upon doing this)

   git rm --cached ./localrepo
   

2. Use git submodule add https//example.com/remoterepo ./localrepo, where ./localrepo is your existing git repository.

   • Note: You get the URL for remoterepo from localrepo/.git/config.
3. Repeat the second step for all your existing repositories that you want to add as submodules.
4. Now you can run git submodule foreach git pull to update all your subprojects.

If you have lots of submodules, you may want to write a small script to automate the second step.

Raid

git rm --cached ./localrepo This was missing for me in other answers. Otherwise I got '<rep>' already exists in the index – Setup