Git submodule update

Question

I'm not clear on what the following means (from the Git submodule update documentation):

...will make the submodules HEAD be detached, unless --rebase or --merge is specified...

How does --rebase/--merge change things?

My main use case is to have a bunch of central repositories, which I will embed via submodules into other repositories. I would like to be able to improve on these central repositories, either directly in their original location, or from within their embedding repositories (the ones that use them via submodule).

• From within these submodules, can I create branches/modifications and use push/pull just like I would in regular repositories, or are there things to be cautious about?
• How would I advance the submodule referenced commit from say (tagged) 1.0 to 1.1 (even though the head of the original repository is already at 2.0), or pick which branch's commit is used at all?

Iridotomy

On the topic of "detached head", see also https://mcmap.net/q/11804/-head-and-orig_head-in-git and stackoverflow.com/questions/237408/… for a practical example (not submodule-related, but still) – Obligate

"you cannot modify the contents of the submodule from within the main project": yes, true. And I have edited my answer to shed some light on that apparent contradiction (non-modifiable submodule, that you still can modify from the main project repo!) – Obligate

Answer 1

This GitPro page does summarize the consequence of a git submodule update nicely

When you run git submodule update, it checks out the specific version of the project, but not within a branch. This is called having a detached head — it means the HEAD file points directly to a commit, not to a symbolic reference.
The issue is that you generally don’t want to work in a detached head environment, because it’s easy to lose changes.
If you do an initial submodule update, commit in that submodule directory without creating a branch to work in, and then run git submodule update again from the superproject without committing in the meantime, Git will overwrite your changes without telling you. Technically you won’t lose the work, but you won’t have a branch pointing to it, so it will be somewhat difficult to retrieve.

---

Note March 2013:

As mentioned in "git submodule tracking latest", a submodule now (git1.8.2) can track a branch.

# add submodule to track master branch
git submodule add -b master [URL to Git repo];

# update your submodule
git submodule update --remote 
# or (with rebase)
git submodule update --rebase --remote

See "git submodule update --remote vs git pull".

MindTooth's answer illustrate a manual update (without local configuration):

git submodule -q foreach git pull -q origin master

In both cases, that will change the submodules references (the gitlink, a special entry in the parent repo index), and you will need to add, commit and push said references from the main repo.
Next time you will clone that parent repo, it will populate the submodules to reflect those new SHA1 references.

The rest of this answer details the classic submodule feature (reference to a fixed commit, which is the all point behind the notion of a submodule).

---

To avoid this issue, create a branch when you work in a submodule directory with git checkout -b work or something equivalent. When you do the submodule update a second time, it will still revert your work, but at least you have a pointer to get back to.

Switching branches with submodules in them can also be tricky. If you create a new branch, add a submodule there, and then switch back to a branch without that submodule, you still have the submodule directory as an untracked directory:

---

So, to answer your questions:

can I create branches/modifications and use push/pull just like I would in regular repos, or are there things to be cautious about?

You can create a branch and push modifications.

WARNING (from Git Submodule Tutorial): Always publish (push) the submodule change before publishing (push) the change to the superproject that references it. If you forget to publish the submodule change, others won't be able to clone the repository.

how would I advance the submodule referenced commit from say (tagged) 1.0 to 1.1 (even though the head of the original repo is already at 2.0)

The page "Understanding Submodules" can help

Git submodules are implemented using two moving parts:

• the .gitmodules file and
• a special kind of tree object.

These together triangulate a specific revision of a specific repository which is checked out into a specific location in your project.

---

From the git submodule page

you cannot modify the contents of the submodule from within the main project

100% correct: you cannot modify a submodule, only refer to one of its commits.

This is why, when you do modify a submodule from within the main project, you:

• need to commit and push within the submodule (to the upstream module), and
• then go up in your main project, and re-commit (in order for that main project to refer to the new submodule commit you just created and pushed)

A submodule enables you to have a component-based approach development, where the main project only refers to specific commits of other components (here "other Git repositories declared as sub-modules").

A submodule is a marker (commit) to another Git repository which is not bound by the main project development cycle: it (the "other" Git repo) can evolves independently.
It is up to the main project to pick from that other repo whatever commit it needs.

However, should you want to, out of convenience, modify one of those submodules directly from your main project, Git allows you to do that, provided you first publish those submodule modifications to its original Git repo, and then commit your main project refering to a new version of said submodule.

But the main idea remains: referencing specific components which:

• have their own lifecycle
• have their own set of tags
• have their own development

The list of specific commits you are refering to in your main project defines your configuration (this is what Configuration Management is all about, englobing mere Version Control System)

If a component could really be developed at the same time as your main project (because any modification on the main project would involve modifying the sub-directory, and vice-versa), then it would be a "submodule" no more, but a subtree merge (also presented in the question Transferring legacy code base from cvs to distributed repository), linking the history of the two Git repo together.

Does that help understanding the true nature of Git Submodules?

Obligate

by publishing you mean commiting? – Iridotomy

@deepblue: yes, pushing. I have added the link to the tutorial which illustrates both the normal workflow and the incorrect one. – Obligate

interesting, the git-submodule man page says that the contents of the git submodule directory cannot be modified "you cannot modify the contents of the submodule from within the main project" – Iridotomy

Wow. That long an explanation for something that is principally that simple ought to be enough to scare any newcomer into just sticking with their svn:externals. ;-) – Neoclassic

@conny: but, as I detail in " Why are git submodules incompatible with svn externals? ", submodules are fundamental different and not compatible with svn:externals. – Obligate

So, if each submodule is tied to the specific commit that was the current one at the time when I added the submodule: how would I go about updating (pull/fetch) that submodule to include the latest changes from upstream? For example, I'm keeping my dotfiles in a repository and inside that, I've added things like vim-plugins as submodules, which I rarely intend to tamper with, but nevertheless want to update on regular intervals. – Caliginous

Sorry, to answer my own question, I gather cd'ing into the submodule and git checkout a sha, or git pull/fetch will do fine. Then commiting the update in the local repository. – Caliginous

@hced: You can also hit all submodules at once using git submodule foreach – Appendage

.. still not getting it. is there an easier explanation on the web somewhere? – Fortney

@Eugene it can help realizing this (git submodules) is really suited for component-based development (https://mcmap.net/q/13003/-what-is-component-driven-development): what is the list of sources (with their exact revision) you need for your program to work? That list of precise point in history (https://mcmap.net/q/13010/-checkout-past-git-submodule-commit) of different groups of sources is represented by git submodules. – Obligate

@IulianOnofrei Since git 1.8.2, git submodule update --remote, provided the submodule is configured to follow a branch. – Obligate

So this answer is incomplete then. – Tenerife

@IulianOnofrei I have edited the answer to make that option more visible. – Obligate

I would delete this answer in favor of his, instead of incorporating it here, as it doesn't seem fair. Also, his comment is the correct solution, not his answer. – Tenerife

@IulianOnofrei no, I would not delete that answer! The all point of this answer is to point out what a submodule actually is. But yes, if you are after the git command to blindly copy-paste, without realizing what you are doing, then MindTooh's answer is the right one for you. – Obligate

Where is a special kind of tree object? – Edwin

@Edwin a gitlink: https://mcmap.net/q/13011/-git-tree-object-and-git-terminology – Obligate

Overengineered answer – Ordonez

Answer 2

To update each submodule, you could invoke the following command (at the root of the repository):

git submodule -q foreach git pull -q origin master

You can remove the -q option to follow the whole process.

Michaelemichaelina

If you just run git submodule update --init --recursive from the root, it will get all of them recursively and initialize them if they aren't already. – Trisa

@SamSoffes That serves a different purpose entirely. Updating submodules will check out the submodules at the commit they are currently pointing to (not necessarily the latest commit). The solution in the above answer updates the commit of each submodule to the latest HEAD from origin/master. – Thickening

My new method: git submodule update --rebase --remote – Michaelemichaelina

Answer 3

To address the --rebase vs. --merge option:

Let's say you have super repository A and submodule B and want to do some work in submodule B. You've done your homework and know that after calling

git submodule update

you are in a HEAD-less state, so any commits you do at this point are hard to get back to. So, you've started work on a new branch in submodule B

cd B
git checkout -b bestIdeaForBEver
<do work>

Meanwhile, someone else in project A has decided that the latest and greatest version of B is really what A deserves. You, out of habit, merge the most recent changes down and update your submodules.

<in A>
git merge develop
git submodule update

Oh noes! You're back in a headless state again, probably because B is now pointing to the SHA associated with B's new tip, or some other commit. If only you had:

git merge develop
git submodule update --rebase

Fast-forwarded bestIdeaForBEver to b798edfdsf1191f8b140ea325685c4da19a9d437.
Submodule path 'B': rebased into 'b798ecsdf71191f8b140ea325685c4da19a9d437'

Now that best idea ever for B has been rebased onto the new commit, and more importantly, you are still on your development branch for B, not in a headless state!

(The --merge will merge changes from beforeUpdateSHA to afterUpdateSHA into your working branch, as opposed to rebasing your changes onto afterUpdateSHA.)

Arakawa

Answer 4

Git 1.8.2 features a new option ,--remote, that will enable exactly this behavior. Running

git submodule update --rebase --remote

will fetch the latest changes from upstream in each submodule, rebase them, and check out the latest revision of the submodule. As the documentation puts it:

--remote

This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote-tracking branch.

This is equivalent to running git pull in each submodule, which is generally exactly what you want.

(This was copied from this answer.)

Tenerife

It's not "yet another answer", as NO other answer has this command (prove me wrong). Other answers didn't work for me, this comment did, so I decided to post it as an answer while giving credit to the original owner. So consider removing your downvote. – Tenerife

There's a comment by MindTooth from 2015 saying this is what they do now. You give no explanation of what this does (though you do mention MindTooth, but there's no real explanation of what you mean by that -— embedding URLs, as in this comment, would help). You don't say why it is a good idea. You don't give any caveats. This isn't, in my view, a helpful answer because it raises more questions than it resolves. – Lianneliao

By that, I meant that it works instead of not working. Believe me, if more people were to see this answer, they would be glad, because it works. For things like this, most people just want to know the command that updates a git submodule, not how it's implemented. – Tenerife

I edited the answer to prove you wrong, also, https://mcmap.net/q/12700/-git-submodule-update/… !!! – Tenerife

You've dramatically improved the answer — it would be sensible to do the editing before making comments so that I see your improvements, not the status quo ante. I'd prefer to see a discussion of the --rebase too. Reassuring that it works is good; explaining what it does helps too. – Lianneliao

But you realise that is was clear for everybody that as of almost 2018, this was the correct command that achieves this simple task, right? – Tenerife

Definitively not "clear for everybody" — I don't realize that at all. I didn't know it. I'm not sure how big of a minority I'm in (if, indeed, I'm in a minority). The answer was not helpful to me because there was zero explanation of what it does. It is better; it still ain't perfect. It ain't my down-vote, so I can't undo it. It is my delete vote; I'm trying to remember/discover how to cancel that. – Lianneliao

A few years later, looking over this comment thread - a perfect example of an overzealous SO editor. Glad the OP of the answer stood up for himself. Wow. – Wotton

@DanNissenbaum, Wow, you're the first person to back me up. And I've taken a lot of downvotes, closed questions/answers, etc. for trying to help others. – Tenerife

"there was zero explanation of what it does" - it obviously updates git submodules, because that's what was asked 😂 – Tenerife