Pushing all submodules recursively

Question

I wrote the below script to push all the changes in the workspace, both in the submodules and the superproject. However, it sounds a little odd that, it is this complex to do what I want. Is there any better way, that I'm missing?

#!/bin/bash

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters"
    exit
fi

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd "${SCRIPT_DIR}/../submodule1" 
git status
git add -A
git commit -m "$1"
git push origin master

cd "${SCRIPT_DIR}/../submodule2" 
git status
git add -A
git commit -m "$1"
git push origin master

cd "${SCRIPT_DIR}/../submodule3" 
git status
git add -A
git commit -m "$1"
git push origin master

printf "\n\nUpdating Super Project\n\n" 
cd .. 
git status
git add -A
git commit -m "All Submodules Updated - $1"
git push origin master

Vidette

Answer 1

git1.7.11 ([ANNOUNCE] Git 1.7.11.rc1) mentions:

"git push --recurse-submodules" learned to optionally look into the histories of submodules bound to the superproject and push them out.

So you can use:

git push --recurse-submodules=on-demand

Gomel

Answer 2

You can use git submodule foreach to run any desired command on each submodule, e.g.

git submodule foreach git push origin master

See: man git-submodule.

Seafood

I believe you forgot the --recursive option for foreach. – Clothier

Answer 3

The command git push --recurse-submodules=on-demand does not work if you have submodules which contain submodules. (git version: 2.20.1)

Add the following alias in your ~/.gitconfig file:

[alias]
    push-all = "! find . -depth -name .git -exec dirname {} \\; 2> /dev/null | sort -n -r | xargs -I{} bash -c \"cd {}; git status | grep ahead > /dev/null && { echo '* Pushing: {}'; git push; }\""

Then issue git push-all in your parent git folder.

Explanation

• !: We are issuing a non-git command.
• find . -depth -name .git -exec dirname {} \\; 2> /dev/null : Find all submodules (and git repositories, which wouldn't harm)
• | sort -n -r: Sort by path depth, deepest will be first.
• | xargs -I{} bash -c \": Pass directories to the following commands:
  • cd {};: cd to the target directory.
  • git status | grep ahead > /dev/null: Test if it is necessary to push this repository.
  • && { echo '* Pushing: {}'; git push; }\"": Inform and push.

Waggery

Answer 4

The git-powercommit script I wrote recently does push submodules recursively as a part of its workflow. Under the hood, there is a map-like iterator which uses git-status --porcelain=v2 to iterate over repository objects. Here is its definition:

mapchanges() {(
  set -e -x
  local filter="$1"
  local commiter="$2"
  git status --no-renames --porcelain=v2 | \
  while read N XY sub mH mI mW hH hI path ; do
    $filter $N $XY $sub $mH $mI $mW $hH $hI "$path"
  done | \
  sort -u --reverse | \
  while read path; do
    $commiter "$path"
  done
)}

In order to iterate over submodules, you need to provide it with the filter and the action callback functions. In your case, the filter function could be:

filter_subm() {
  # Inputs are according to `git status --porcelain=v2` spec. The function
  # filters submodules which has changed commits.
  local sub="$3"; shift 8; local path="$1"
  case "$sub" in
    SC??) echo "$path" ;;
    *) ;;
  esac
}

As for the action function, the origina script does commit the whole submodule, but in your case you could insert the push commands like the following

push_subm() {
  local path="$1"
  (cd -P "$path" && git push origin; )
}

Now, we bring everything together with the line like

mapchanges filter_subm push_subm

Please, consider reviewing the original script to find out the details on how to organize the recursion.

Vogul

Answer 5

This c# console app prompts you for each submodule with changes and allows you to easily add/commit/push or run custom git commands:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;

static class Program
{
    static readonly Process Proc = new Process();

static void Main(string[] args)
{
    Console.WindowHeight = (int) (Console.LargestWindowHeight * .8f);
    var dir = Path.GetFullPath(args.Length == 1 ? args[0] : Directory.GetCurrentDirectory());

    foreach (var module in GetModules(dir))
    {
        Console.WriteLine("Processing: " + module);
        PrintDiff(module, out var untracked, out var mods);

        if (!untracked && !mods)
            continue;

        while (true)
        {
            Console.WriteLine("Use \"c <msg>\" to commit and push" + (untracked ? ", \"a <msg>\" to also add untracked files" : "") + ", r to refresh.\nTo fix issues use git as normal, or cmd to enter prompt (exit to leave)" + ":");

            var cmd = Console.ReadLine();

            if (cmd.StartsWith("a "))
                Run("git", "add .", module);

            if (cmd.StartsWith("a ") || cmd.StartsWith("c "))
            {
                Run("git", "commit -a -m \"" + cmd.Substring(2) + "\"", module);
                Run("git", "push", module);
                break;
            }

            if (cmd.Trim() == "r")
            {
                PrintDiff(module, out untracked, out mods);
                if (!untracked && !mods)
                    break;
                continue;
            }

            try
            {
                var parts = cmd.Split(' ');
                Run(parts[0], cmd.Substring(parts[0].Length), module);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Console.WriteLine();
            }
        }

        Console.WriteLine();
    }

    Console.WriteLine("Press any key to continue...");
    Console.ReadKey();
}

static void PrintDiff(string module, out bool untracked, out bool mods)
{
    var u = false;

    Run("git", "ls-files . --exclude-standard --others", module, s =>
    {
        u = true;
        Console.WriteLine("Untracked file: " + s);
    });

    var m = false;
    Run("git", "diff --staged", module, t =>
    {
        m = true;
        if (t.StartsWith("diff --git"))
            Console.WriteLine();
        Console.WriteLine(t);
    });

    Run("git", "diff", module, t =>
    {
        m = true;
        if (t.StartsWith("diff --git"))
            Console.WriteLine();
        Console.WriteLine(t);
    });

    if (m)
        Console.WriteLine();

    if (!u && !m)
    {
        Console.WriteLine("No changes");
        Console.WriteLine();
    }

    untracked = u;
    mods = m;
}

static List<string> GetModules(string dir)
{
    var l = new List<string>();
    var paths = new Queue<string>();
    var modules = new List<string>();
    paths.Enqueue("");

    while (paths.Count > 0)
    {
        var d = Path.Combine(dir, paths.Dequeue());
        if (!File.Exists(Path.Combine(d, ".gitmodules")))
            continue;

        Run("git", "config --file .gitmodules -l", d, t =>
        {
            var parts = t.Split('=');
            if (parts[0].EndsWith(".path"))
                l.Add(Path.Combine(d, parts[1]));
        });

        foreach (var s in l)
            paths.Enqueue(s);
        modules.AddRange(l);
        l.Clear();
    }

    modules.Reverse();
    return modules;
}

static void Run(string fileName, string arguments, string directory)
{
    Run(fileName, arguments, directory, Console.WriteLine);
    Console.WriteLine();
}

static void Run(string fileName, string arguments, string directory, Action<string> a)
{
    Proc.StartInfo = new ProcessStartInfo
    {
        FileName = fileName,
        Arguments = arguments,
        RedirectStandardOutput = true,
        WorkingDirectory = directory,
        UseShellExecute = false
    };
    Proc.Start();
    string t;
    while ((t = Proc.StandardOutput.ReadLine()) != null)
        a(t);
}
}

Londonderry