Managing many git repositories

Question

Setting up a project is easy in git and so I can have separate repository even for small script. Now the problem is how to manage them.

I work in multiple places with these repositories. When I have done changes to some repository, I want to be able to update the repositories in other places.

So I have a directory with many repositories in it.

1. How can I fetch all of them?
2. How can I check whether any of them have uncommitted changes?
3. How can I check whether any of them have changes to merge?

And it would be nice to be able to do these with one command.

The output needs to be silent enough to actually notice the things to do.

Rhondarhondda

The same question answered for hg mercurial. – Tallou

I use multi-cursor-select/multi-line editing feature of my code editor (VS code) to create batch/shell script and execute it in one go. With this simple stupid solution, I know what I'm doing and what I can expect from execution of those commands. No guessing, no learning, no customisation needed. Another reason is, every time I want to execute commands on different set of repos which reside in a common parent directory. – Hep

Note: with Git 2.30 (Q4 2020), you now have the new git for-each-repo command – Scarrow

github.com/earwig/git-repo-updater — “gitup is a tool for updating multiple git repositories at once. It is smart enough to handle several remotes, dirty working directories, diverged local branches, detached HEADs, and more. It was originally created to manage a large collection of projects and deal with sporadic internet access. gitup should work on macOS, Linux, and Windows. You should have the latest version of git and either Python 2.7 or Python 3 installed.” – Nathalienathan

tsrc is inspired by Google AOSP git-repo. I find it's much simpler yet powerful. It also eliminates so many issues that git-repo has on Windows. I highly recommend it. Check out the comparison to similar tools on its doc. – Philippines

@Scarrow the big disadvantage of git for-each-repo is that prints the results of the commands, but doesn't say which results corresponds to which repo – Selfcommand

@Selfcommand You should be able to run git rev-parse --git-dir for your command sequence in a git for-each-repo loop: that would display each repository root folder. – Scarrow

@Scarrow how do I run several git commands in one git for-each-repo loop? – Selfcommand

@Selfcommand You can run git for-each-repo --config=<config> -- xxx args: that will look for a git-xxx executable in $PATH, in which you can put as many comma d as you want. – Scarrow

Answer 1

I highly recommend the multiple repositories tool mr. I used to use a custom shell script as recommended by others for some time, but using mr has the following benefits for me:

• It's generic: A conjunction of various version control systems can be used, not only git (e.g. Mercurial, SVN, etc.).
• It's fast: mr can execute multiple jobs in parallel. I use a number of git/mercurial repositories and sync them several times a day. Mr tremendously speeds up this process.
• It's easy and quick to manage the list of repository checkouts. Just use 'mr register' rather than modifying the list of projects in your custom script.

Regarding to your question about silent output: The level of verbosity can be modified using the command line switch -q. I prefer the default output which appears to nicely unify the output in a short and clear summary.

I use the following alias for the mr command to ensure that mr always picks up my default project list stored in $HOME, and uses 5 parallel threads:

alias mr='mr -d ~/ -j 5 '

Spacial

it's great but does not support git tagging (as of 2016-08) – After

@CADbloke it doesn't mention windows in the install page, but as a Perl script it might/should/could work on windows. – Weir

@HugoZaragoza, you can always define this command by yourself [DEFAULT] tag = git tag "$@" – Commorancy

alias pa='find . -name .git -print -execdir git pull \;' then, pa is enough – Infer

is this tool free ? @Sandro Giessl any drawbacks of this ? – Treacle

Answer 2

I must say I started with the currently accepted answer (just a bunch of helpers scripts that iterate over the repositories), but all in all, it was a lacking experience for me.

So, after trying mr, repo and git-submodules, I found each lacking in a different way, so, I ended up doing my own variant: http://fabioz.github.io/mu-repo which is a mature tool at this point -- it has workflows which allow you to:

• clone multiple repos
• diff (and edit) current changes in multiple repos
• preview incoming changes
• run commands in multiple repos in parallel
• create groups of repos
• run non git-related commands over multiple repos
• etc (see homepage for more info).

Note that it supports any OS where Python runs ;)

Tuberosity

Your answer suggest that mu-repo is better than what mr does. But mu-repo doesn't support non git repos. – Newhouse

Whell, it's purpose is really to support only git repos (which is what the question asked) -- although you can do mu sh <some command> to execute some arbitrary command in multiple repos too, its whole finality is dealing with git and not other version control systems... if you need that, then yes, please use a different tool. – Tuberosity

Answer 3

gr (git-run) extends mr's functionality (only for git). I find it easier to organize multiple git repos using its tag system. The code for gr is not well maintained though. If you are using bash, make sure you use it with the -t tag instead of #tag form.

Crown

It looks like development has picked back up a bit since then: github.com/mixu/gr/graphs/contributors – Whipcord

Answer 4

I wrote a commandline tool called gita to manage multiple repos. It shows the status of registered repos side by side, for example

It can also delegate git commands/aliases from any working directory.

Now to answer your questions using this tool:

How can I fetch all of them?

gita fetch

How can I check whether any of them have uncommitted changes?

The gita ll command shows 3 possible symbols next to the branch name, which indicates

• +: staged changes
• *: unstaged changes
• _: untracked files/folders

How can I check whether any of them have changes to merge?

The branch names are colored in 5 ways

• white: local branch has no remote branch
• green: local branch is the same as remote branch
• red: local branch has diverged from remote branch
• purple: local branch is ahead of remote branch (good for push)
• yellow: local branch is behind remote branch (good for merge)

To install it, simply run

pip3 install -U gita

Purser

Answer 5

I wrote a tool called "RepoZ" which automatically discovers git repositories as soon as you clone them or change anything in them like switching a branch, for example.

Once found, the repositories are "tracked". That will simply show them in a list of local repositories including a dense status information inspired by posh-git. This contains the current branch and further stuff like file edits and the count of incoming or outgoing commits.

This helps to keep track of your local repositories and unfinished work to commit or push. In addition, you can use the repository list as navigation to switch from one repository to another.

"How can I fetch all of them?"

The version for Windows offers a context menu to fetch or pull a repository. With multi-select you can run actions on multiple repositories at once.

However, you might find another feature very useful:

With Auto fetch, you can tell RepoZ to fetch the remotes of all your git repositories periodically in the background. These fetches won't collide with your local commits, of course. There are no local merge attempts like with git pull.

Overtire

Yeah, too bad and not deserved. Open for PRs at any time. Ping me. – Overtire

I'm a Windows user with a lot of repos and this tool is pretty good. – Annoyance

Answer 6

You could try using repo with a custom manifest.xml file to specify where your repositories are. There is some more documentation on how to do this.

It's the standard for Android AOSP project.

Alternatively you could use git-submodule(1).

Shall

git-submodule would be good, if I would want to keep the repositories at the exact same state, but that is not the case. The set of repositories is not the same in each place. There could be uncommitted changes. There could be changes I don't want to merge yet. – Rhondarhondda

I didn't know about git-submodule. Thanks for mentioning it. – Kentledge

Documentation for repo is quite minimal and looks like it is intended for different kind of work flow I want to use. – Rhondarhondda

this is the standard for Android AOSP huge project. Still don't seams easy to create a custom manifest.xml or anything not related to AOSP anyhow... – Treadway

Answer 7

I've made an alias and a function to run any git command on all repositories available in a directory (recursively). You can find it here: https://github.com/jaguililla/dotfiles/git

This is the code:

#!/bin/sh
# To use it: source git_aliases

# Example: rgita remote \;
alias rgita='find . -type d -name .git -execdir git'

# Example: rgit remote -vv
rgit() {
  rgita "$@" \;
}

Hope it helps :)

Profusive

slim and useful ! – Disallow

Fantatstic! Simple one-liner to put in .bash_aliases: alias rgit='function _rgit(){ find . -type d -name .git -printf "\n%p\n" -execdir git "$@" \;; }; _rgit'. Source it, then e.g. call rgit status. – Loft

Answer 8

I use this script to easily execute git commands in all of my repositories.

#!/bin/sh
if [ ! "$1" = "" ] ; then

   if [ "$GITREPO" = "" -a -d "$HOME/cm/src" ] ; then
      GITREPO="$HOME/cm/src"
   fi

   if [ "$GITREPO" != "" ] ; then

      echo "Git repositories found in $GITREPO"
      echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-"

      DIRS="`/bin/ls -1 $GITREPO`"

      for dir in $DIRS ; do

         if [ -d $GITREPO/$dir/.git ] ; then
            echo "$dir -> git $1"
            cd $GITREPO/$dir ; git $@
            echo
         fi

      done
   else

      echo "Git repositories not found."

   fi
fi

By default the script will look for git repositories in ~/cm/src but you can override this by setting the GITREPO environment variable to your liking.

This script is based on this script.

Peddling

find . -name .git -print -execdir git pull \; does what you think. – Infer

@Infer your pithy comment seems like the best answer to me! (Although it looks like the script in this answer also echo the pull call?) – Navicert

Answer 9

gitslave is a tool which can run the same command over many repositories by creating a superproject/subproject relationship between the super and the subs. This (by default) provides output summarization so you can concentrate on the repositories which provide unique output (useful for git status, not so useful for git ls-files).

This is typically used for projects where you need to assemble several repositories together and keep them on the same branch or tag at the same time or whatever. For my directory of (bare) repositories I just have a little makefile which lets me run arbitrary git commands, which as you see I primarily use for fsck and gc:

full: fsck-full gc-aggressive
        @:

fsck-full:
        for f in */.; do (cd $$f; echo $$f; git fsck --full || echo $$f FAILED); done

gc-aggressive:
        for f in */.; do (cd $$f; echo $$f; git gc --aggressive || echo $$f FAILED); done

%:
        for f in */.; do (cd $$f; git $@ || echo $$f FAILED); done

Archaism

Answer 10

You can use the git-status-all gem for this: https://github.com/reednj/git-status-all

# install the gem    
gem install git-status-all

# use the status-all subcommand to scan the directory
git status-all

There is also a fetch option that will fetch from origin for all the repositories before displaying the status:

git status-all --fetch

Hymenopteran

Very nice approach. In contrast to most of the higher voted tools, there is no need to first register repositories, instead it simply checks all subdirectories. – Otherworld

find . -name .git -print -execdir git status \; is OK – Infer

Answer 11

For this purpose, you can use every which can run cli commands in multiple directories following this pattern every <command>

Install: npm i -g every-cli

Usage example: every git branch

- repo-1
  master
- repo-2
  develop

Abdullah

Answer 12

If anyone is still looking at this thread please checkout my shell script called gitme

you will need to manually input your local git repos but i use this tool everyday to collaborate multiple git projects on multiple computers.

you can run git clone https://github.com/robbenz/gitme.git

also the script is posted below

#!/bin/bash -e

REPOS=( 
/Users/you/gitrepo1
/Users/you/gitrepo2
/Users/you/gitrepo3
/Users/you/gitrepo4
)

MOVE="Moving to next REPO... \n" 

tput setaf 2;echo "What ya wanna do? You can say push, pull, commit, ftp push, or status"; tput sgr0

read input

if [ $input =  "commit" ]
then
    tput setaf 2;echo "Do you want a unique commit message? [y/n]";tput sgr0
    read ans
    if [ $ans = "y" ]
    then 
        for i in "${REPOS[@]}"
        do
            cd "$i"
            tput setaf 6;pwd;tput sgr0 
            git add . -A
            read -p "Commit description: " desc  
            git commit -m "$desc"
            tput setaf 2;echo  $MOVE;tput sgr0 
            sleep 1
        done 
    else 
        for i in "${REPOS[@]}"
        do
            cd "$i"
            tput setaf 6;pwd;tput sgr0 
            git add . -A
            git commit -m "autocommit backup point"
            tput setaf 2;echo  $MOVE;tput sgr0 
            sleep 1
        done
    fi 
elif [ $input = "push" ] || [ $input = "pull" ] || [ $input = "ftp push" ] || [ $input = "status" ]
    then
        for i in "${REPOS[@]}"
do
    cd "$i"
    tput setaf 6;pwd;tput sgr0 
    git $input 
    tput setaf 2;echo  $MOVE;tput sgr0 
    sleep 1
    done 
else tput setaf 1;echo "You have zero friends";tput sgr0 
fi

I setup an alias in my ~/.bash_profile so alias gitme='sh /path/to/gitme.sh'

Zanze

Answer 13

I've found a great solution for pulling from the current branch from all sub-directories that have a .git folder, even if each repo has a different branch checked out. The command is a one-liner, flexible enough to modify for different git commands, and can be aliased.

Just copy and paste the following:

find . -type d -maxdepth 2 -name .git -exec sh -c 'cd "$0" && cd .. && git pull origin $(git rev-parse --abbrev-ref HEAD)' {} \;

Breaking it down:

find . -type d -maxdepth 2 -name .git 

Find all directories (-type d) in the current directory (find .) that have the name ".git" (-name .git), looking a maximum of two directories deep (2 rather than 1 because we're looking for the git folder within the git repo folder).

-exec sh -c 

Run the following shell command (exec sh -c)

'cd "$0" && cd .. && git pull origin $(git rev-parse --abbrev-ref HEAD)'

Change directory to the first argument (cd "$0"), then change directory one level up to leave the .git folder (cd ..) then do git pull origin while specifying the branch by running git rev-parse... for the current branch's HEAD commit.

{} \;

The "{}" is the result relative path we get from the initial find command. The ; is used to end the command.

Tested on MacOS 10.14.6 on zsh. As-is, works only when remote and local branches are named the same, AFAIK.

You can modify this to get status. I expect you might be able to add arguments to further make this more flexible but I haven't tried yet.

Zischke

Answer 14

I wrote a CLI tool for this, see https://manicli.com/. It also has some nice additional features:

• Run commands over many repos
• Declarative configuration
• Flexible filtering, select 1 project, projects having a tag, or a subpath

Surrender

Answer 15

You should check out rgit on the CPAN which recursively executes git commands on all repositories in a directory tree.

From the docs:

This utility recursively searches in a root directory (which may be the current working directory or - if it has been set - the directory given by the GIT_DIR environment variable) for all git repositories, sort this list by the repository path, chdir into each of them, and executes the specified git command.

Ramburt

Answer 16

I have just created a tool that do what you want!

1. How can I fetch all of them? gmaster fetch
2. How can I check whether any of them have uncommitted changes? gmaster status
3. How can I check whether any of them have changes to merge? gmaster status

It is called gitmaster: https://github.com/francoiscabrol/gitmaster

Hepzi

Answer 17

I wrote this simple bash function into my .bash_profile to be able to run git, maven, gradle, or any other bash command in all git repositories only:

# usefull function to work with multiple git repositories
all() {
    failed=0
    printf '>>> START RUNNING: "%s" >>>' "$*"
    for d in */; do
        pushd "$d" > /dev/null
        if [ -d ".git" ]
        then
            printf '\n--------------------------------------------\n\n'
            pwd
            eval $@
            if [ $? -ne 0 ]
            then
                failed=1
                break;
            fi
        fi
        popd > /dev/null
    done
    if [ $failed -eq 0 ]
    then
        printf '\n--------------------------------------------\n'
        printf '<<< RAN "%s" WITH SUCCESS!!! <<<' "$*"
    else
        printf '\n<<< FAILED!!! <<<'
    fi
}

This can be used this way

all git st
all git pull
all ./gradlew clean test
etc.

Infatuate

one could also run them all in parallel using: all './gradlew clean test &' – Infatuate

Answer 18

Disclaimer: I am working on this tool at www.repoflow.com

How can I fetch all of them?
How can I check whether any of them have changes to merge?

• You can fetch all the involved repositories (or push/pull/commit them), after the fetch the UI will be updated with the new repository state, if the repository is diverged you can see the kind of conflicts and begin merge them.

How can I check whether any of them have uncommitted changes?

• On the UI you can see the state of the files for each repository (You can add/remove them individually or in bulk).

I am working on this, but this is also solving the OP questions and help manage multiple repositories in general, you can use the UI or the underlying GraphQL API to create your own scripts, please consider it as another option.

Laocoon

Answer 19

There's a handy tool to fetch all multiple repositories. git-pull-all is a command line tool to execute on multiple git repositories in asynchronously.

Installation

npm install -g git-pull-all

Usage

Assume you have these files and directories:

~/Projects/
  cool-examples/
    .git/
  funny-movies/
  my-todos.txt
  super-express/
    .git/

When you run git-pull-all command on ~/Projects directory, it should find child git repositories (in the above case cool-examples and super-express) then execute git pull on each of them.

$ cd ~/Projects
$ git-pull-all
funny-movies/
Not a git repository
cool-examples/
Already up-to-date.
super-express/
Already up-to-date.
Done!

git-pull-all ~/Projects

---

GitHub link: https://github.com/tatsuyaoiw/git-pull-all

Moire

Answer 20

I use Visual Studio code. We have around 20 libraries in our codebase that are all separate Git repos, and the source control tab in Visual Studio Code is pretty good for seeing an "at a glance" view of how far behind or ahead local repos are. There are also settings to periodically fetch to keep that info up to date, as well as a refresh button.

Its not as convenient as a script, but when it comes to source control for me I like to be the one making changes. It takes a very well-written script to be careful to not overwrite changes, etc. With the VS Code approach, you get a good amount of info quickly that you can act on yourself.

Here is a screenshot of what it looks like:

Messmate

Answer 21

Expanding on https://mcmap.net/q/181287/-managing-many-git-repositories (Now repo is available on GitHub)

I suggest you check this tool https://github.com/googleapis/github-repo-automation used by Google to manage multiple repos.

About your question about dedicating changes, changes should be PRs so you can react to them.

https://github.com/tj/git-extras could help you write a script for your scenarios.

1. How can I fetch all of them? repo sync on the google solution

2. How can I check whether any of them have uncommitted changes? You can easily write your logic using the lib feature

3. How can I check whether any of them have changes to merge? PR should be the ultimate indication that branches need to merge, but if you have another logic - the above

check git extras commands for ideas see https://github.com/tj/git-extras/blob/master/Commands.md#git-show-unmerged-branches

Wagonette

Answer 22

Please try gitas.

gitas status flags:

  -t, --time           time of last commit shown (default true)
  -f, --format {r|i}   format time: relative|iso (default r)
  -b, --branch         branch shown
  -q, --query          query fetch needed (implies -br)
  -r, --remote         remote shown
  -l, --url            url shown
  -d, --dirty          dirty shown (default true)
  -u, --untracked      untracked shown
  -s, --stash          stash shown
  -o, --order {t|n}    order: time|name (default t)
  -e, --emit {t|j|m}   emit format: table|json|markdown (default t)
  -h, --help           help for status

Kenogenesis

Answer 23

I use multi-cursor-select/multi-line editing feature of my code editor (VS code) to create batch/shell script and execute it in one go. With this simple stupid solution, I know what I'm doing and what I can expect from execution of those commands. No guessing, no learning, no customisation needed. Another reason is, every time I want to execute commands on different set of repos which reside in a common parent directory.

Hep

Answer 24

Please follow below-mentioned steps to pull code from all the repositories:

1. create a new file using extension as .bat
2. place that file in the same hierarchy where you have all your repository folders
3. edit .bat file and add below-mentioned script

FOR /D %%G in (%cd%\*) Do cd %%G & call git branch & call git pull & cd ..
echo All code pulled successfully!!
pause

4. save the file and double click to start pulling your code

Rosalia

Answer 25

https://github.com/wwjamieson3/envisionTools has a bash script called gitstat that does this and a whole lot more. It's GNU and mac compatible. It can perform fetches from remotes if asked. It shows untracked, modified, added, deleted and staged files. It differences with remotes by showing unmerged commits on remotes and unpushed local commits. It also can display color coded results in summary or detailed mode and even HTML. It uses locate instead of find because it's infinitely faster and will even prompt to update your locate database if it's getting too old.

Gravesend

Thanks for posting your answer! Please be sure to read the FAQ on Self-Promotion carefully. Also note that it is required that you post a disclaimer every time you link to your own site/product. – Gnathonic

Unfortunately, the github wwjamieson3/envisionTools link is broken. – Unscathed

@williamJamieson is envisionTools is a private GitHub repository? – Illeetvilaine