Share a working tree with git submodules

Question

Suppose I have a library Common that may be used stand-alone, and is used by projects P1 and P2, so the tree I want looks like

/Common/.git
        ...
/P1/.git
    .gitmodules  # points to remote server
    Common/
    ...
/P2/.git
    .gitmodules  # points to remote server
    Common/
    ...

When I make a change in /Common, I would like to be able to test it using P1 and P2 before committing. With the usual git submodule command set, I would have to commit from /Common, push to remote, then pull from both /P1/Common and /P2/Common. If the commit breaks something, it cannot be amended because the bad change has already been published. Alternatively, I could git remote add quicktest /Common from /P?/Common to be able to pull without touching the remote server. But this has lots of opportunities for inconsistency and it is dirty to strip broken commits from /P?/Common so that they can be amended in /Common.

I would rather that, during development, the working tree from /Common be used by P1 and P2, but I can't make /P1/Common a symlink to /Common because git submodule recognizes the symlink as different from a directory. Hardlinking directories is not allowed by most file systems. I can hardlink all the files using

rm -rf /P1/Common
cp -rl /Common /P1/Common

which works quite well until a new file is added to /Common in which case this process needs to be repeated. Is there an elegant way to both

1. keep git clone --recursive git://remote/P1.git working for the end user, and
2. allow me to easily test that changes in /Common work with P1 and P2?

Runofthemine

Try git worktree feature(since git 2.5) on your submodule. And see this answer – Gamecock

Answer 1

I would rather establish an intermediate bare repo where to:

• push Common new commits
• pull from for P1/Common and P2/Common

At least, if commits are amended/removed later, that intermediate repo has never been published outside and you still can reset your P1/Common and P2/Common submodules to that intermediate repo content.

---

Update: if you are using git worktree, since Git 2.5 (July 2015, 5 years after writing this answer), make sure to use Git 2.25 (Q1 2020).
Before that, "git worktree add" internally calls "reset --hard", which affected submodules.

Usurp

I don't see what this intermediate repo buys me over pulling from the non-bare /Common. I still have to commit to test, and if the test fails, it needs to be cleaned up. – Runofthemine

@Jed: what it buys you is the possibility to force push to it, rewriting the history without any consequences for any other teammates. – Usurp

Compare git clone --bare /Common /Common-bare; cd /P1/Common; git remote add quicktest-bare /Common-bare to git remote add quicktest /Common. The bare repository doesn't seem to add anything, neither involves a remote. But both of these have several steps to test a patch. Cleaning out /Common-bare can be done with git push -f, but the extra indirection doesn't seem to add anything. And, the point of my question is that I want to reduce the number of steps, not increase the number. – Runofthemine

@Jed: then add Common (the local one, where you commit) as remote from P1/Common and P2/Common and pull directly from there. – Usurp

Yes, that was described in the sentence "Alternatively, I could git remote add quicktest /Common from /P?/Common to be able to pull without touching the remote server." The main problem is that cleaning up the clones at /P1/Common and /P2/Common is messy and easy to forget. In particular, suppose that a change worked in P1, but not in P2. After fixing the "obvious" typo that was causing the failure, one might commit --amend in /Common, then pull -f from /P2/Common and retest P2. Assuming this passes, it would be natural to push /Common, commit P1 and P2, ... – Runofthemine

... and push. You might have noticed that /P1/.gitmodules now points to a revision that does not exist. Maybe this is turning into a new feature, but I think it would be desirable to have /P1/Common as a symlink to /Common without git treating the symlink as a normal file (as far as submodule versioning is concerned). I wonder why git submodule refuses to follow the symlink. – Runofthemine

@Jed: Why don't have alias for those pulls? (alias that would pull from P1 and P2 each time) – Usurp

I.e. shell script? I dislike putting destructive operations in scripts, mostly because of error handling and (in other related cases) composability reasons. But I think you are right that a shell script is adequate in this case. Note that a shell script could also just cp -rlTf /Common /P1/Common and then committing would not even be required and edits of existing files would automatically be visible (without even running the above). – Runofthemine

@Jed: note that alias can also be pure git commands, which would be compatible in any case. – Usurp

Answer 2

Try git worktree feature since git 2.5.

1. Delete /P1/Common
2. cd /Common
3. Create P1 branch for the coming /P1/Common working tree
4. run git worktree add ../P1/Common P1

Do the same thing on /P2.

Then, /P1/Common, /P2/Common and/Commonworking trees share the same repository/Common/.git`.

And you can easily just checkout any commit which is commit in another working tree.

P.S. you can still use git submodule command for daily work with this git worktree feature.

Gamecock