Checkout git submodule from azure pipeline

Question

I am currently working on an azure pipeline. Within my main github repo (repo A), I have another github repo added as a sub module. (repo B)

My goal is to checkout the sub-module at the start of the pipeline with the following YAML:

stages:
- stage: checkout
  jobs:
  - job: checkout
    steps:
      - checkout: self
        submodules: true
        persistCredentials: true

This then attempts to checkout the sub-module, but ends with the following error:

Cloning into '/home/vsts/work/1/s/devops-scripting'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
fatal: clone of 'https://github.com/sourcerepo/devops-scripting.git' into submodule path '/home/vsts/work/1/s/devops-scripting' failed

It seems to be an issue with using an incorrect user/password - if i was pushing i could simply use supply user/pass parameters, however this doesn't seem to work for checking out.

How can i update a submodule via an azure pipeline?

Tortuous

Answer 1

Checkout git submodule from azure pipeline

If the github repo (repo A) and sub module (repo B) are in the same GitHub organization, then the token stored in the GitHub service connection is used to access the sources.

If not, you can instead use a custom script step to fetch submodules. First, get a personal access token (PAT) and prefix it with pat:. Next, base64-encode this prefixed string to create a basic auth token. Finally, add this script to your pipeline:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: basic <BASE64_ENCODED_TOKEN_DESCRIBED_ABOVE>" submodule update --init --recursive

Be sure to replace "<BASIC_AUTH_TOKEN>" with your Base64-encoded token.

Use a secret variable in your project or build pipeline to store the basic auth token that you generated. Use that variable to populate the secret in the above Git command.

Another workaround, Use custom script to reuse access token for submodule sync:

steps:
- checkout: self
  submodules: false
  persistCredentials : true

- powershell: |
    $header = "AUTHORIZATION: bearer $(System.AccessToken)"
    git -c http.extraheader="$header" submodule sync
    git -c http.extraheader="$header" submodule update --init --force --depth=1

Check more info from this post.

Deth

This is beyond obtuse. Does Azure really expect people to figure this out? On the other hand, why is submodules set to true by default. Like if it wouldn't fail... – Deannadeanne

Answer 2

Below method supports checking out submodules in the different ADO projects located in the same ADO organization. Be aware that, all submodules MUST be on https!

- script: |
    git -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" \
        submodule update --init --recursive
  displayName: Fetch Submodules

Dinge

Answer 3

Check if you can allow your Azure script to access the system token, which is used authenticate against VSTS from inside your builds and releases

steps:
- checkout: self
  persistCredentials: true

This is illustrated in this answer.

I suppose the issue arises because of the nature of the submodule repository: if it is a private repository, it would need credentials to be cloned.

Flanker