How to use submodules publicly, but symlinks to a single clone locally?

Question

I'm having my first Git Submodule experience.

I have some projects that depend on the same subproject. I keep these projects in sync, so I'm using the "submodule branch" feature (e.g. git submodule add -b master [URL]).

While I'd like the public GitHub repositories to convey the submodule relationship, in my own workflow I'd really just like to have one clone of the shared codebase on my disk. I thought I could just set up the submodules, and then do a switcheroo with a symbolic link. But when I do, I get this:

On branch master
Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git checkout -- <file>..." to discard changes in working directory)

    typechange: draem

So Git apparently sees the fact that it's a symbolic link, instead of following through to the directory.

Is there any workflow where I can appear to be working with submodules, but really only have one clone on my local filesystem?

V

You probably already know this, but this approach is a tad perilous. You risk fixing library code for one application only to break it in another, even for very small projects. This is exactly what submodules are intended to help you avoid. When each client gets its own versioned (and possibly even branched) version of the dependency, merging and resolving conflicts becomes an explicit activity that prevents these mistakes. Also, patching the submodule for applications can be done on hotfix- branches whose merger can be deferred or even omitted -- eliminating conflict with mainline code. – Rimose

Answer 1

So Git apparently sees the fact that it's a symbolic link, instead of following through to the directory.

Yes, Git would see such a change, because that submodule is declared in the parent repo as a special entry in the index.

Making a symlink would replace that special entry by a file of another type.

What you could do is try playing with GIT_WORK_TREE (as in "Including submodules in git checkout to GIT_WORK_TREE in hook").

But a simpler solution would be to:

• keep your submodule right where they are.
• add another clone of that submodule repo where you want it (/path/to/sub).
• detect any changes from the original submodule folder with a git --work-tree=/path/to/sub status from within your duplicated submodule folder in your parent repos.

It will detect changes between:

• the submodule repository HEAD (the original submodule folder, where you execute the git --work-tree=/path/to/sub status command)
• the working tree you are designated (--work-tree=/path/to/sub)

You can:

• work from your submodule cloned repository outside your main parent repository (Ie.: where you want)
• detect changes done in that outside folder right from your submodule folder of your main parent repository (the one which references all your submodules)

Bernstein

Thanks...the detect changes and hooking thing sounds like a good answer for more complex projects. But for me at this stage I really need something simple. I've found things are seeming to work okay by making that clone on the side and linking it, and then just not checking the submodule in for the commits. Hacky, but it's seeming to work (if I'm not missing anything...) – V

Can you explain more the "detect any changes" part? I don't follow how to implement this. Thank you. – Lissettelissi

Answer 2

You can bind mount the submodule's other directory

sudo mount --bind /path/to/main/repo relative/path/to/submodule

I can't find much info about this method, but saw it listed in this, similar question.

Eckmann

Answer 3

(Appreciate the other advice, but answering with what I've decided to do, because it's easiest.)

Put clones of the submodules you are planning to share in their own directories.

Then set things up how you like in the using projects, each with their own clone of the submodule(s) you are planning to share. Once the projects are set up in a way that new users would get the submodules normally, replace the submodule directories with shared symbolic links to your single local instance where you do your edits, and add them to the .gitignore.

(Rather than just deleting the submodule-based directories, you might want to rename them out of the way. You could then move them back into place and pull them if necessary.)

So the working style is to just never commit the submodule / typechange. As long as you're willing to always track the master branch, people cloning the project will still get the right thing. Meanwhile, you can work with the single repository on your disk.

For projects at a more advanced stage, @VonC probably has the right answer... to not lie to Git, check in the submodules at the appropriate states, and use triggers to manage the updates.

V

Interesting approach. +1 – Bernstein

Please explain how i do that in easiest way. Currently suffering from exact same problem .I am not able to get Get it checked in. ? – British

@AbhishekAryan "checked in" is old programmer language for "commit and push". I've added a bit more detail to this approach. It's not fancy, and a bad idea for serious work--which should synchronize to specific commits on submodules instead of always pulling the master. More a convenience for someone working alone at an early stage of project development. – V

Answer 4

Since it hasn't been mentioned before.
One could also override local tracking logic for specific git entry.

git update-index --assume-unchanged path/to/submodule

This will ignore any local changes to single file and works with submodules (as submodule is not treated as directory by git).

If needed, one can also simply re-enable local tracking again.

git update-index --no-assume-unchanged path/to/submodule

This solution scales nicely for multiple submodules in separate directory, as one could setup script to enable/disable exclusion of entire submodules directory, like mentioned in this answer.

Reprobation

This is the best answer here. I made a script wrapped around this idea that lets me toggle between the real submodule and a separate checkout. – Railhead

Answer 5

I personally wouldn't advice to have submodules of different projects point to the same code. If the code from the submodule is really really big, it could be an argument, but in most situations I wouldn't do it.

This can also be good because you can develop in the submodule of one of your projects and not affect the other ones.

Godiva

Thanks, though in my case it's actually more of a hassle to deal with the multiple synchronizations. Although I can see having different project states providing a benefit in a more complicated project worked on by multiple people... for me at this project's state of maturity it's just added overhead. – V

It might be very useful to avoid creating a lot of commits of partial code just make sure changes build. Something you could avoid with a squash merge of course but still easy to work with. – Monopetalous

Answer 6

In theory you could also edit main-project/src/submoduleA/.git. It would be a text file like containing this:

.git/modules/src/submoduleA

Perhaps git would not care if you changed that path to something like:

.git/../../submoduleA

I'm not sure if git would allow this, or if some git versions would choke while others work. Since this edit would not be tracked via git, I am not even going to try it out.

Perhaps git would allow a submodule as a sibling directory... lets test that:

clientA ➤ git submodule add https://github.com/myco/utils ../utils
The following path is ignored by one of your .gitignore files:
../utils
Use -f if you really want to add it.
clientA ➤ git submodule add -f https://github.com/myco/utils ../utils
fatal: ../utils: '../utils' is outside repository
'../utils' is not a valid submodule name

Cant even fool git, I have the latest on mac os (2.24.1)

git submodule add -f https://github.com/myco/utils src/../../utils
fatal: ../utils: '../utils' is outside repository
'../utils' is not a valid submodule name

A symlink is a file. A submodule, src/submoduleA needs to be a folder that contains a .git text file as described above.

Marileemarilin

I would share this as a comment, but I can't really do multi-lines or format code snippets – Marileemarilin

Given this understanding how submodules work, the concept of a symlinked submodule starts to fall apart, or at least becomes a little shaky. Submodules are fully designed to be git repos inside another git repo. The .git file stores a reference to another git repo inside .git/modules. I tried adding a submodule as a sibling to the current git repo I was in (clientA). Git refused to do this. But more hacking could get something to work. Perhaps the .git file could be manually edited post-clone or something (in an npm preinstall/postinstall script) – Marileemarilin