Automatically access git submodules via ssh or https

Question

Question:
Is there a way to automatically checkout git submodules via the same method (ssh or https) as the main repository?

Background:

We have a non-public gitlab repository (main) that has a submodule (utils) which is also hosted as a non-public gitlab repository on the same server. Those repositories can be accessed either via ssh or https:

• [email protected]:my/path/repo.git
• https://gitlabserver.com/my/path/repo.git

Both variants obviously require different forms of authentication and depending on the client computer and the user, one or the other is preferred.

For the top level repository (main) that is not an issue, as anyone can choose the method he or she prefers, but for the sub module this depends on the .gitmodules file and hence is (initially) the same for all.
Now instead of everyone having to adapt the .gitmodules file to whatever they prefer and make sure they don't accidentally commit those changes, it would be nice, if there was a way to just specify the server and repo path and git chooses either the same method that is used for the main repo, or something that can be set in gitconfig.

Eyra

tldr sorry. but I use git-hooks to run git submodule foreach git submodule update or similar. I have no time to write a proper answer but i hope this will help. – Dartmouth

Answer 1

I finally solved this problem by specifying the submodules url as a relative path:

So lets say your main git repository can be reached

• either via https://gitlabserver.com/my/path/main.git
• or via [email protected]:my/path/main.git

And the .gitmodules file looks like this:

[submodule "utils"]     
    path = libs/utils   
    url = https://gitlabserver.com/my/path/utils.git

That would mean that even when you check out the main application via ssh, the submodule utils would still be accessed via https.

However, you can replace the absolute path with a relative one like this:

[submodule "utils"]     
    path = libs/utils   
    url = ../utils.git

and from now on use

• either git clone --recursive https://gitlabserver.com/my/path/main.git
• or git clone --recursive [email protected]:my/path/main.git

to get the whole repository structure which ever way you want. Obviously that doesn't work for cases where the relative ssh and the https paths are not the same, but at least for gitlab hosted repositories this is the case.

This is also handy if you (for whatever reason) mirror your repository structure at two different remote sites.

Eyra

This also won't work if a module points to a different repository, right? For example if this is the central project [email protected]/abc/win/win-agent.git and this one a referenced module [email protected]/abc/mob/module.git – Unprintable

@David: I can't test it, but I don't see, why it wouldn't work (obviously you'd have to use ../../mob/module.git in this case). – Eyra

I tested it with ../ updating doesn't fail though but it doesn't seem logical for me, it shouldn't work, it should only work with ../../ as you wrote. – Unprintable

Ok, following worked: root project [email protected]/scm/win/win-agent.git referencing module [email protected]/scm/mob/module.git with following config: [submodule "modules/module"] path = modules/module url = ../../mob/module.git – Unprintable

Thank you for this, one additional thing I had to, after altering the .gitmodules, do was run; git submodule sync (reference; dhoeric.github.io/2017/https-to-ssh-in-gitmodules) – Bankroll

This is not a general solution, because the naming pattern of the ssh and https urls might not match. But in the case that was haunting the the last few days, this trick worked perfectly, and I bet it does quite widely, so THANKS! – Malvinamalvino

I just used git submodule add ../utils.git libs/utils and it worked. No need to change if later. – Drier

March 2021, and this does not work for me. Still no permission to clone the submodule, with relative path. It works flawlessly in my own terminal emulator. – Maine

This works well, but may cause some grief with forks. Since it's a relative path, git will look for a fork of each of the submodules as well, eg in github.com/your-user/REPO. Every developer who works on a fork will have to fork and maintain each submodule. – Nedranedrah

@Nedranedrah you can use ../../username/Repo this way it will still use the repository from the original user in forks – Presber

Answer 2

I think I have a more general answer, but it's not quite automatic. There's a configuration setting you can apply in ~/.gitconfig that will replace a particular URL with the one of your choosing.

For Github I have this in my ~/.gitconfig:

[url "ssh://[email protected]/"]
    insteadOf = https://github.com/

Here's a repository with a submodule declared with an HTTPS url that I can clone recursively using HTTPS.

git clone --recursive https://github.com/p2/OAuth2.git

OR

git clone --recursive [email protected]:p2/OAuth2.git

You can read about its use here: https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf

And here: https://git-scm.com/docs/git-clone

Master

Thanks. That setting looks quite useful. I' d still prefer to use relative paths where possible, because I can just tell someone else to recursively clone repository X without further manual configuration. However, it seems perfect from the other end: I want to clone someone else's repository tree that doesn't use relative paths with a different method (or mirror) than the author intended. – Eyra

And if course you can't always work with relative paths. – Eyra

Sure thing. I noticed this doesn't work super well with XCode and Swift package manager, but neither do relative submodules. – Master

Unfortunately this only seems to work with the argument passed to git clone. It doesn't work with the submodules themselves. – Cureton