Checkout multiple git repos into same Jenkins workspace

Question

Using Jenkins 1.501 and Jenkins Git plugin 1.1.26

I have 3 different git repos each with multiple projects.

Now I need to checkout all projects from the 3 git repos into the same workspace on a Jenkins slave. I have defined each git repo in: Source code Management: Multiple SCMs. But each time a repo is checked out the previous repo (and its associated projects) is deleted.

I have read this:

http://jenkins.361315.n4.nabble.com/multiple-git-repos-in-one-job-td4633300.html

but its does not really help. I have tried to specify the same folder under Local subdirectory for repo (optional) for all repos but it gives the same result.

If this is simply impossible using Jenkins I guess some pre-build step/scripting could be used to move the projects into the right location. Its not an option to modify the build configuration of the projects.

Migration

Answer 1

Checking out more than one repo at a time in a single workspace is not possible with Jenkins + Git Plugin.

As a workaround, you can either have multiple upstream jobs which checkout a single repo each and then copy to your final project workspace (Problematic on a number of levels), or you can set up a shell scripting step which checks out each needed repo to the job workspace at build time.

Previously the Multiple SCM plugin could help with this issue but it is now deprecated. From the Multiple SCM plugin page: "Users should migrate to https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin . Pipeline offers a better way of checking out of multiple SCMs, and is supported by the Jenkins core development team."

Norvil

Why is the first approach problematic? Splitting up jobs seems like good practice. – Sleep

It is a good practice in general, but when you need multiple checkouts in the same physical location maintenance becomes a huge concern. For instance, if you wanted to create a branch build, you would have to clone 4 jobs and then individually change the paths for each one. There are of course plugins to help with this, but it's easier to just checkout to a relative path from a single job. Then you can clone as much as you want without changing settings. – Norvil

This would also be a problem if you wanted to run builds in parallel as the workspace would not be in a fixed location. – Norvil

You need to change it from the correct answer as it's not relevant anymore. – Springing

The physical location wouldn't matter if you use the Copy Artifact plugin. I use that extensively across slaves with no problems. – Ballew

Pipelines require you to learn a new DSL, which is a little bit too much for the very simple job (check out code from multiple repositories) we want it to do. Stick to the Multiple SCM plugin until a decent GUI appears around the Jenkins pipeline DSL. I can report that it's been working fine with Jenkins 2.17 – Zennas

I just ran into the same issues with multiple repos. I am now using the Pipeline plugin even though I was just as skeptical as @BurakArslan about the new DSL. It's actually not as bad as I thought and comes with a reasonably decent snippet generator. After using it for only 2 hours I now actually prefer this approach since I can eventually commit the pipeline build scripts to git together with the rest of the code. – Caine

Answer 2

With the Multiple SCMs Plugin:

• create a different repository entry for each repository you need to checkout (main project or dependancy project.

• for each project, in the "advanced" menu (the second "advanced" menu, there are two buttons labeled "advanced" for each repository), find the "Local subdirectory for repo (optional)" textfield. You can specify there the subdirectory in the "workspace" directory where you want to copy the project to. You could map the filesystem of my development computer.

The "second advanced menu" doesn't exist anymore, instead what needs to be done is use the "Add" button (on the "Additional Behaviours" section), and choose "Check out to a sub-directory"

• if you are using ant, as now the build.xml file with the build targets in not in the root directory of the workspace but in a subdirectory, you have to reflect that in the "Invoke Ant" configuration. To do that, in "Invoke ant", press "Advanced" and fill the "Build file" input text, including the name of the subdirectory where the build.xml is located.

Autostrada

This must be outdated. At the time of writing Multiple SCMs plugin GIT snippet does not contain the optional sub-path. – Corker

In each repository there is a drop down list called "Add". In it you can find the option "Checkout to a sub-directory", which performs the same. – Encroachment

I followed your guide with the multiple SCM plugin and git but I have another funny problem. It seems to not want to properly checkout the same branch (develop) for different repositories. It tries to checkout a commit by hash (which is valid only in the first repository). Any idea on how to address this? – Exercitation

The biggest issue with multiple SCM plugin is this: "Post-commit type triggers don't currently work (at least for subversion), so it is necessary to configure 'cron' type polling." – Dettmer

I don't know whether works fine in git or not. I set up the jobs to poll from the repos, and every time it detects a commit to ANY of the repos involved in the job, it triggers the job – Autostrada

Doesn't work for me. Followed all the instructions apparently successfully, but the second checkout fails - irrespective of which repo I put first or second. Something "deep git technical" wrong: fatal: reference is not a tree: f33e270ec039f0002a309ea1820f8c9a4b35ab97 – Plagal

> Deprecated: Users should migrate to wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin. Pipeline offers a better way of checking out of multiple SCMs, and is supported by the Jenkins core development team. – Showiness

Pipelines require you to learn a new DSL, which is a little bit too much for the very simple job (check out code from multiple repositories) we want it to do. Stick to the Multiple SCM plugin until a decent GUI appears around the Jenkins pipeline DSL. I can report that it's been working fine with Jenkins 2.17 – Zennas

Answer 3

Since Multiple SCMs Plugin is deprecated.

With Jenkins Pipeline its possible to checkout multiple git repos and after building it using gradle

node {   
def gradleHome

stage('Prepare/Checkout') { // for display purposes
    git branch: 'develop', url: 'https://github.com/WtfJoke/Any.git'

    dir('a-child-repo') {
       git branch: 'develop', url: 'https://github.com/WtfJoke/AnyChild.git'
    }

    env.JAVA_HOME="${tool 'JDK8'}"
    env.PATH="${env.JAVA_HOME}/bin:${env.PATH}" // set java home in jdk environment
    gradleHome = tool '3.4.1' 
}

stage('Build') {
  // Run the gradle build
  if (isUnix()) {
     sh "'${gradleHome}/bin/gradle' clean build"
  } else {
     bat(/"${gradleHome}\bin\gradle" clean build/)
  }
}
}

You might want to consider using git submodules instead of a custom pipeline like this.

Foetus

Thank you!!! The dir block is key, I couldn't figure out why I was only seeing the most-recent-cloned repo in my job's workspace. – Quinquereme

how is the notion of "changes" considered under multiple SCM's? Is it just the sum of the changes seen from all repos that make up the jobs? Would be good to itemize them from each if possible, 23 changes from repo XXX, 3 changes from repo YYY or something more compact along those lines. – Postfix

I'm using git submodules but I need to change branch when a new PR is created on any of the multiple repos.. any idea how to achieve that? – Turnstile

@ArnoldRoa Not sure about your setup. Lets assume you have github, there you would have a webhook event when a pr is created - in that event the branch of that pr is supplied within the event (or at least some kind of ref). So you can simply take the branch name of that event. Havent, tried it, though. – Foetus

Answer 4

I used the Multiple SCMs Plugin in conjunction with the Git Plugin successfully with Jenkins.

Plausible

thanks that's great, i am able to put 2 bitbucket paths into the repository section, and now how can i tell repo 1 checkout "develop" branch and for the repo 2 checkout "fixes" branch? i see the branches to build portion in jenkins, how can i set up the repository name and Refsec in the branch specifier (blank for 'any') so each one can be check out the corresponding branch that i want? or i am doing it wront and i should click the boolean that say "Multiple SCMs"? – Furze

@Furze Were you able to find the solution ? – Appal

at the moment we not using the yml file and we did the two different workflows with the regular tasks – Furze

Answer 5

Jenkins: Multiple SCM - deprecated. GIT Plugin - doesn't work for multiple repos.

Scripting / pipeline as code - is the way to go.

Smashup

Answer 6

Depending upon the relationships of the repositories, another approach is to add the other repository (repositories) as a git submodules to one of the repositories. A git submodule is creates a reference to the other repos. Those submodule repos are not cloned unless the you specify the --recursive flag when cloning the "superproject" (official term).

Here's the command to add a submodule into the current project:

git submodule add <repository URI path to clone>

We are using Jenkins v1.645 and the git SCM will out-of-the-box do a recursive clone for superprojects. Voila you get the superproject files and all the dependent (submodule) repo files in their own respective directories in the same Jenkins job workspace.

Not vouching that this is the correct approach rather it's an approach.

Zoography

How to change submodules to a different branch? I want to setup jenkins to run test on any branch from where a PR is created. – Turnstile

Answer 7

I also had this problem. I solved it using Trigger/call builds on other projects. For each repository I call the downstream project using parameters.

Main project:

This project is parameterized
String Parameters: PREFIX, MARKETNAME, BRANCH, TAG
Use Custom workspace: ${PREFIX}/${MARKETNAME}
Source code management: None

Then for each repository I call a downstream project like this:

Trigger/call builds on other projects: 
Projects to build: Linux-Tag-Checkout
Current Build Parameters
Predefined Parameters: REPOSITORY=<name>

Downstream project: Linux-Tag-Checkout:

This project is parameterized
String Parameters: PREFIX, MARKETNAME, REPOSITORY, BRANCH, TAG
Use Custom workspace:${PREFIX}/${MARKETNAME}/${REPOSITORY}-${BRANCH}
Source code management: Git
git@<host>:${REPOSITORY}
refspec: +refs/tags/${TAG}:refs/remotes/origin/tags/${TAG}
Branch Specifier: */tags/${TAG} 

Rabb

Answer 8

Checking out more than one repo at a time in a single workspace is possible with Jenkins + Git Plugin (maybe only in more recent versions?).

In section "Source-Code-Management", do not select "Git", but "Multiple SCMs" and add several git repositories.

Be sure that in all but one you add as an "Additional behavior" the action "Check out to a sub-directory" and specify an individual subdirectory.

Factious

I believe in this way you are using the (deprecated) Multiple SCM plugin rather than the vanilla Git Plugin. – Pessa

Answer 9

We are using git-repo to manage our multiple GIT repositories. There is also a Jenkins Repo plugin that allows to checkout all or part of the repositories managed by git-repo to the same Jenkins job workspace.

Uncrowned

How exactly do you solve the problem asked in this question? I've installed the plugin you mentioned, and read about repo and the plugin, but I can't see how to set up Jenkins to clone two repos to execute in one project... – Plagal

In order to use Repo, the special repository need to be created which will contain only the manifest file(s). In this file you specify all the information about other repositories. The exact format of manifest files is described in the docs/manifest-format.txt file of the git-repo project (gerrit.googlesource.com/git-repo/+/master/docs/…). When configuring Jenkins Repo part of the job - you specify the location of the 'manifest' repository and optionally the name of the 'manifest' files (you may have several). All repositories specified in the manifest will be cloned. – Uncrowned