Checkout submodules in Jenkins with Github organisation plugin

Question

I have a build job in Jenkins created by the Github Organization plugin. The Jenkinsfile for this build checkouts the code using checkout scm which is good as it figures out the correct branch/revision to checkout when building either PR triggered changes or pushes to the master branch.

How can I make this:

node {
  checkout scm 
}

checkout submodules?

Fifield

Answer 1

The solution with sh 'git submodule...' works only for Repositorys without special authentication.

We use following solution in our set up:

node {
    checkout([
        $class: 'GitSCM',
        branches: scm.branches,
        doGenerateSubmoduleConfigurations: true,
        extensions: scm.extensions + [[$class: 'SubmoduleOption', parentCredentials: true]],
        userRemoteConfigs: scm.userRemoteConfigs
    ])
}

Hugibert

That does tell Jenkins to attempt git submodule update but I get Permission denied (publickey). Beats me why as submodule update works when configured via the UI. – Stanch

This answer probably comes from this article: support.cloudbees.com/hc/en-us/articles/… – Stanch

Another way to find this syntax is to use the Snippet Generator under Pipeline Syntax. In my case that generates a snippet similar to but somewhat larger than the above which appears to address repository access but also does not work. – Stanch

@DaveSchweisguth using parentCredentials: true solved the Permission denied (publickey) issue you're seeing, assuming you have stored your SSH credentials in Jenkins. – Remembrancer

I also had to approve the three scripts referred to here: issues.jenkins-ci.org/browse/JENKINS-42860 one at a time until all three were approved. Their error messages in the build output were quite explicit. After that, it is working like a champ. The code snippets form within Jenkins were getting me in the ballpark, but not as completely as this answer. – Breast

I see a similar behaviour as @DaveSchweisguth does: Even with parentCredentials: true, the submodule checkout still fails (though the parent checkout works). I use a GitHub access token (stored in Jenkins as username/password credentials). – Hydrography

And I just found out why: We use SSH to checkout code, but the GitHub Organisation plugin by default uses HTTPS for code checkouts. But the submodule checkout uses whatever was recorded when the submodules were added, in our case an SSH address. Thus, the parentCredentials were not used. Find the hidden "Checkout over SSH" additional behaviours. – Hydrography

/job/ooxx-build/pipeline-syntax/ I use above path in my Jenkins server to auto generate pipeline. – Beset

Answer 2

In the Github organization plugin, add the advanced submodule behaviors.

And configure it like this:

---

As @oeuftete pointed out, you also may need to add the "Checkout over SSH" behaviour (and provide a key credential) if the submodule(s) use the SSH protocol.

As documented here: https://wiki.jenkins.io/display/JENKINS/Git+Plugin

Duval

This works, but you also may need to add the "Checkout over SSH" behaviour (and provide a key credential) if the submodule(s) use the SSH protocol. – Azal

It works, but it assumes that all repos in the project can live with the same sub-modules behaviour. This should usually be the case, but corner cases might not be able to make this work – Gleanings

For me I had to also purge the existing checkout on the build machine: the submodule had been checked out differently before. – Leiva

Why would this work if the submodule requires OTHER credentials than the main repo? – Cohesive

Answer 3

Change it to this:

node {
    checkout scm
    sh 'git submodule update --init'
}

Use bat instead of sh if Jenkins is running on Windows.

Shawanda

This works for me, but only if the Jenkins user's default SSH key can read all of the repos being updated. Per my comment on the other answer I could not get the credentials stored in Jenkins to work for submodules, even though they work for the primary checkout. Also, because I have multiple submodules, the SSH key can't be a deploy key (which can only be associated with one repo) but needs to be associated with a Github user with appropriate permissions. – Stanch

Answer 4

We had similar issue, Jenkin user is using https to pull from Github but the submodule is using SSH and we want to handle the pull requests with Jenkins. I did the below checkout stage hope it will help someone here:

stage('Checkout') {
  if(env.BRANCH_NAME == "develop" || env.BRANCH_NAME == "master") {
    checkout([
      $class: 'GitSCM',
      branches: scm.branches,
      doGenerateSubmoduleConfigurations: false,
      extensions: [[$class: 'SubmoduleOption',
                    disableSubmodules: false,
                    parentCredentials: true,
                    recursiveSubmodules: true,
                    reference: '',
                    trackingSubmodules: false],
                   [$class: 'CleanBeforeCheckout'], 
                   [$class: 'CleanCheckout']],
      submoduleCfg: [],
      userRemoteConfigs: [[credentialsId: 'jenkins-ssh',
                           url: '[email protected]:<AccountName>/<RepoName.git>']]
                          ])
  }
  else if (env.CHANGE_ID) {
    checkout([
      $class: 'GitSCM',
      branches: [[name: "FETCH_HEAD"]],
      doGenerateSubmoduleConfigurations: false,
      extensions: [[$class: 'SubmoduleOption',
                    disableSubmodules: false,
                    parentCredentials: true,
                    recursiveSubmodules: true,
                    reference: '',
                    trackingSubmodules: false],
                   [$class: 'CleanBeforeCheckout'], 
                   [$class: 'CleanCheckout']],
      submoduleCfg: [],
      userRemoteConfigs: [[credentialsId: 'jenkins-ssh', 
                           refspec: "+refs/pull/${CHANGE_ID}/head:refs/remotes/origin/${BRANCH_NAME} +refs/heads/${CHANGE_TARGET}:refs/remotes/origin/${CHANGE_TARGET}", 
                           url: '[email protected]:<AccountName>/<RepoName.git>']]
    ])
  }
}

Maybe there is an easier way to do it, I would be happy to hear from you :-)

Retreat

There seems to be an easier way: Configure the GitHub Organisation plugin to do the checkout using SSH, unless you need to use HTTPS to pull from GitHub. Note that the GitHub Organisation plugin requires HTTPS to scan the repo, but not to checkout. – Hydrography

Yes thank you for your comment. There is also another way... In the Github organization plugin, add "Checkout over SSH" and select or add the SSH key. – Retreat