Automatic initialization and update of submodules in Makefile

Question

I have a git project with several sub-modules (a choice I somewhat regret in retrospect). It uses gnu make

Currently I expect people to issue by hand a git submodule update --init¹ before building the project for the first time and also after pulling any change that updated a submodule reference.

However, I'd like for the Makefile to issue these commands automatically when they are needed. It's OK if they issue the commands sometimes when they aren't needed (spurious update) - but it shouldn't happen regularly.

For the initial init it seems enough to have a rule like (for a submodule that lives in directory module1:

module1/.git:
    git submodule update --init

And here the choice of .git as the file to "represent" the submodule is fairly arbitrary, it could be some other file instead.

However this doesn't work that well to update the submodule when the reference has been updated. I guess I could make the submodule depend on the root .gitmodules file which I guess should change when a submodule reference gets updated, something like:

module1/.git: .gitmodules
    git submodule update --init

Here the use of .git seems wrong though: presumably that directly won't necessarily be updated when update is run (especially if there was no update to this particular submodule), which will leave the update command running every time.

Looking for a cleaner solution here.

---

¹ Or possibly use the --recursive argument on the initial clone, which has the same effect.

Springing

You could probably use .git/modules/module1/HEAD as a prerequisite to update "module1". Or to use git submodule status to create list of submodules to update (or you could also create a makefile that would update only changes submodules). – Hemp

If I used .git/modules/module1/HEAD as the pre-req, what would be the target (presumably something inside the module)? – Springing

Hmm.. good question. First I assumed that module1/.git could be used, but that's not working. And I don't feel using git internal structure a good idea. – Hemp

@Kuchara's answer below works, however there are some philosophical things. You've already discovered that git submodules are a crime against nature and an affront to humanity. Consider alternatives. I also caution against your Makefile mucking with your working copy. It's like crossing the streams in ghost busters - you want to keep those concerns separate. The Makefile is source, don't have it manipulate the source. Consider having your continuous integration system handle cloning the repo as desired. In closing, I recommend Kuchera's solution but have it instruct them to init. – Attaway

@Attaway - I agree partly, but not totally with you. make is the build system, and it is common in general for build systems to download external and perhaps build external artifacts, as necessary. In my case submodules are essentially external artifacts. So I don't see it as unusual that make would update these modules as necessary. In the same way, I don't see it as weird that make would generate source, as in many projects with source generation. I agree it is not desirable for make to "muck with your working copy", but in principle there is a line between the source for ... – Springing

... my project and the submodules. Modifying the submodules is odd enough that I'm find for it to be a totally separate workflow not related to downloading and building my project. I didn't understand at all "Consider having your continuous integration system handle cloning the repo as desired". What CI system are you referring to here? My primary concern is individual users downloading and using my project. I have CI set up, but this isn't a problem as it always gets fresh copies of the submodule using clone --recursive. – Springing

Or maybe you mean that CI can essentially clone the external projects somewhere an inject them in a way the main project can use without using submodule? – Springing

Answer 1

I've crafted something like this:

.PHONY: check-and-reinit-submodules
check-and-reinit-submodules:
    @if git submodule status | egrep -q '^[-+]' ; then \
            echo "INFO: Need to reinitialize git submodules"; \
            git submodule update --init; \
    fi

It is using git submodule status, to figure out if submodule is not initialized (- at the beginning) or somehow modified/outdated (+ at the beginning). This make goal will run always, but git submodule update --init will run only when needed.

Hemp

Note that you also need to add check-and-reinit-submodules to your all target dependenices. Like: all: check-and-reinit-submodules .... – Rainer

...or add another layer of makefiles (overlying makefile would call check-and-reinit-submodules, and run real makefiles). Then no need to modify build system makefiles at all (overlying makefile could be run as make -f other-makefile). – Hemp

Tip: The grep / regex command can be simplified to: egrep -q '^[-+]' – Living

Answer 2

You could generate a file before updating the submodules:

all: .gitmodules_updated my-target

.gitmodules_updated : .gitmodules 
     touch .gitmodules_updated
     git submodule update --init

If .gitmodules is updated it becomes newer than .gitmodules_updated and hence the submodules are updated.

Iolenta

This doesn't work because .gitmodules is only updated when a module is added or removed, not when the commit associated with a gitmodule changes. – Springing