git submodule foreach - Robust way to recursively commit a child module first?

Question

Is there a robust way to do a recursive depth-first git submodule foreach command? I am using the foreach --recursive command which does the job, except it is breadth-first. This is a problem because if I have the following structure:

• A
  • B
• C

And I have commits in all three, a foreach --recursive add -A && git commit ... will hit A, B, C, which is problematic if I want the supermodule to capture the commits of B at that time.

I found this discussion from 2008, but it does not look like any of the suggested features are in the current version of Git that I have (1.7.9.5).

I wrote a small bash function to do this (excuse the shorthand naming):

function git-sfed() { git submodule foreach "git submodule foreach '$*' && $*"; }

And testing it with the following fanciful command seems to work:

git-sfed 'python -c "import sys; print sys.argv" $path'

Does this command seem robust, or are there other common existing methods?

Toussaint

Note: some commands now are aware of submodule: for instance, git grep -e "bar" --recurse-submodules is available with Git 2.12: https://mcmap.net/q/693084/-how-to-git-grep-the-main-repository-and-all-its-submodules – Puggree

Answer 1

You can try this

git submodule  foreach --recursive  |  tail  -r | sed 's/Entering//' | xargs -I% cd % ; git add -A \& git commit

This list (recursively) all the submodules , then reverse the list, tail -r so you get the directories in the order you want (child first), enter the directory and do what ever you want in it.

Nottage

Interesting, more complete answer than mine. +1 – Puggree

Thanks for this one! Unfortunately, though, I did not have the -r option (Ubuntu), but through this post there's the tac command. So with a different flavoring of xargs, came up with another variant: git submodule foreach --recursive | tac | sed 's/Entering //' | xargs -n 1 bash -c 'cd $1 && git status' _ – Toussaint

I've been trying out my method some, but I believe I get an error if I want to run a commit on all submodules and then the supermodule. I will be testing your method later to see if it is more robust. I will come back to it once I test it out. – Toussaint

Sorry it took a while, but I've been having trouble with by yours and my command when trying to put in single quotes (kinda sucks not being able to write them) - escaping with multiple levels of bash commands is a little confusing haha. Right now, I'm figuring the next best option is to try and submit a patch for git-submodule, so I will be trying that out. – Toussaint

Answer 2

I didn't find any other way than your function to perform a depth-first foreach command.

The test would be to check if it does achieve recursive for a depth of more than one.

A
  B
    D
  C

---

I've been having trouble with by yours and my command when trying to put in single quotes (kinda sucks not being able to write them) - escaping with multiple levels of bash commands is a little confusing.

This (quotes issue) should be simplified in Git 1.9/2.0 (Q1 2014), with commit 1c4fb13 from Anders Kaseorg (andersk):

'eval "$@"' creates an extra layer of shell interpretation, which is probably not expected by a user who passes multiple arguments to git submodule foreach:

 $ git grep "'"
 [searches for single quotes]
 $ git submodule foreach git grep "'"
 Entering '[submodule]'
 /usr/lib/git-core/git-submodule: 1: eval: Syntax error: Unterminated quoted string
 Stopping at '[submodule]'; script returned non-zero status.

To fix this, if the user passes more than one argument, execute "$@" directly instead of passing it to eval.

Examples:

• Typical usage when adding an extra level of quoting is to pass a single argument representing the entire command to be passed to the shell.
  This doesn't change that.
• One can imagine someone feeding untrusted input as an argument:

    git submodule foreach git grep "$variable"

That currently results in a nonobvious shell code injection vulnerability.
Executing the command named by the arguments directly, as in this patch, fixes it.

---

Since Git 2.21 (Q2 2017), you have git grep -e "bar" --recurse-submodules

Puggree

I tested out both mb14's and my techniques and they both seem to work. I posted an example on pastebin. – Toussaint

Answer 3

Your method actually will only do two levels of depth. To implement the full recursion, you'll want to call your script itself recursively. Something like the below should do...

#!/bin/bash
SCRIPT=$(realpath "$0")
SCRIPTDIR=$(dirname "$SCRIPT")
git submodule foreach "$SCRIPT '$1' && $1"

Antinucleon