How do I enable my Azure pipeline to checkout a submodule using Git?

Question

In our Git project, we have the following defined in .gitmodules

[submodule "DBPkg"]
  path = DBPkg
  url = https://[email protected]/MainCo/OurProject/_git/DBPkg

When building the Azure Pipeline, in my "Get Sources" pipeline step (I'm using "Azure Repos Git"), I have enabled checking out of submodules

In our project settings, I have also made the following configurations ...

However, when building the project, it dies when attempting to checkout the submodule wiht the following ...

2020-10-16T20:29:44.9912145Z HEAD is now at e06abce Modified docker file to remove unnecessary container name.
2020-10-16T20:29:44.9913415Z ##[command]git submodule sync
2020-10-16T20:29:44.9920288Z ##[command]git -c http.https://[email protected]="AUTHORIZATION: bearer ***" submodule update --init --force
2020-10-16T20:29:44.9923206Z Submodule 'DBPkg' (https://[email protected]/MainCo/OurProject/_git/DBPkg) registered for path 'DBPkg'
2020-10-16T20:29:44.9923842Z Cloning into '/home/vsts/work/1/s/DBPkg'...
2020-10-16T20:29:44.9924246Z remote: TF401019: The Git repository with name or identifier DBPkg does not exist or you do not have permissions for the operation you are attempting.
2020-10-16T20:29:44.9924850Z fatal: repository 'https://dev.azure.com/MainCo/OurProject/_git/DBPkg/' not found
2020-10-16T20:29:44.9925716Z fatal: clone of 'https://[email protected]/MainCo/OurProject/_git/DBPkg' into submodule path '/home/vsts/work/1/s/DBPkg' failed
2020-10-16T20:29:44.9926288Z Failed to clone 'DBPkg'. Retry scheduled
2020-10-16T20:29:44.9926718Z Cloning into '/home/vsts/work/1/s/DBPkg'...
2020-10-16T20:29:44.9927132Z remote: TF401019: The Git repository with name or identifier DBPkg does not exist or you do not have permissions for the operation you are attempting.
2020-10-16T20:29:44.9928020Z fatal: repository 'https://dev.azure.com/MainCo/OurProject/_git/DBPkg/' not found
2020-10-16T20:29:44.9928801Z fatal: clone of 'https://[email protected]/MainCo/OurProject/_git/DBPkg' into submodule path '/home/vsts/work/1/s/DBPkg' failed
2020-10-16T20:29:44.9929412Z Failed to clone 'DBPkg' a second time, aborting

What else do I need to do to give permissions to the pipeline step to checkout the submodule? The URL in the .gitmodules is directly cut and pasted from the Repos section where we would normally clone that repository.

Edit: The URLs for the repos in questions as copied from the portals are

https://[email protected]/MainCo/OurProject/_git/MainAPIs
https://[email protected]/MainCo/OurProject/_git/DBPkg

So the submodule lives at https://[email protected]/MainCo/OurProject/_git/DBPkg but not sure how to phrase the above in terms of a Git submodules.

Edit 2:

In response to the answer given, here is the error output reported by the agent ...

...
2020-10-20T13:21:23.9543308Z ##[command]git submodule sync
2020-10-20T13:21:23.9892501Z ##[command]git -c http.https://[email protected]="AUTHORIZATION: bearer ***" submodule update --init --force
2020-10-20T13:21:24.0147532Z Submodule 'DBPkg' (https://[email protected]/MainCo/OurProject/_git/DBPkg) registered for path 'DBPkg'
2020-10-20T13:21:24.0194148Z Cloning into '/home/vsts/work/1/s/DBPkg'...
2020-10-20T13:21:24.1694357Z remote: TF401019: The Git repository with name or identifier DBPkg does not exist or you do not have permissions for the operation you are attempting.
2020-10-20T13:21:24.1696067Z fatal: repository 'https://dev.azure.com/MainCo/OurProject/_git/DBPkg/' not found
2020-10-20T13:21:24.1726598Z fatal: clone of 'https://[email protected]/MainCo/OurProject/_git/DBPkg' into submodule path '/home/vsts/work/1/s/DBPkg' failed
2020-10-20T13:21:24.1731117Z Failed to clone 'DBPkg'. Retry scheduled
2020-10-20T13:21:24.1760958Z Cloning into '/home/vsts/work/1/s/DBPkg'...
2020-10-20T13:21:24.3885945Z remote: TF401019: The Git repository with name or identifier DBPkg does not exist or you do not have permissions for the operation you are attempting.
2020-10-20T13:21:24.3892068Z fatal: repository 'https://dev.azure.com/MainCo/OurProject/_git/DBPkg/' not found
2020-10-20T13:21:24.3902720Z fatal: clone of 'https://[email protected]/MainCo/OurProject/_git/DBPkg' into submodule path '/home/vsts/work/1/s/DBPkg' failed
2020-10-20T13:21:24.3909036Z Failed to clone 'DBPkg' a second time, aborting
2020-10-20T13:21:24.3987155Z ##[error]Git submodule update failed with exit code: 

1

Isogonic

Do u know if the path /home/vsts/work/1/s/DBPkg exists. If not, create the DBPkg folder in it. – Palingenesis

Answer 1

Update2

Pipelines can access any Azure DevOps repositories in authorized projects, as described in the previous Limit job authorization scope to current project section, unless Limit job authorization scope to referenced Azure DevOps repositories is enabled.

With this option enabled, you can reduce the scope of access for all pipelines to only Azure DevOps repositories explicitly referenced by a checkout step in the pipeline job that uses that repository.

Try to turn off this option and check again.

If it's still not work, you need to check repo's security. Find your [project name] build service account and project collection build service account, make sure both of them have enough permission to access your two git repos.

---

Update

Try to use this in .gitmodules since submodule repo is in the same url as the parent. :

[submodule "DBPkg"]
  path = DBPkg
  url = ../DBPkg

You could also check this link: https://mcmap.net/q/13732/-git-submodule-url-not-including-username

---

The build pipeline will check out your Git submodules as long as they are:

• Unauthenticated: A public, unauthenticated repo with no credentials required to clone or fetch.

• Authenticated:

  1. Contained in the same project, GitHub organization, or Bitbucket Cloud account as the Git repo specified above.

  2. Added by using a URL relative to the main repository. For example, this one would be checked out: git submodule add /../../submodule.git mymodule This one would not be checked out: git submodule add https://dev.azure.com/fabrikamfiber/_git/ConsoleApp mymodule

You could also take a look at the Azure DevOps documentation here.

A similar blog for your reference: Using Git submodules in (private) Azure DevOps repositories

Diurnal

For point #2, "git submodule add /../../submodule.git", I think my question is what does the ".gitmodules" file need to look like? I edited my quesiton to include the URLs of the repos, which are identical except for the last parts. – Isogonic

I would blindly try ../DBPkg ? – Spitball

Hi @PatrickLu-MSFT, I tried your suggestion and included the error output as another edit to my question. I'm fixated on this "DBPkg does not exist or you do not have permissions for the operation you are attempting" message. Is there somewhere where I'm supposed to be setting checkout permissions for the pipeline? – Isogonic

Hi Dave, please check my update reply. Hope it works now. – Diurnal

Hi, Yup, unchecking "Limit job authorization scope to referenced Azure DevOps repositories is enabled" combined with the new .gitmodules format you listed did the trick! – Isogonic

Answer 2

If you don't have permission to change the "Limit job authorization scope to referenced Azure DevOps repositories", adding a checkout step for the submodule repository allowed me to give it permission, and the the submodule worked. It downloads the repository twice though.

- checkout: git://project/RepositoryReferencedByTheSubmodule
- checkout: self
  submodules: true

The URL doesn't need to match, the submodule can point to something like https://[email protected]/xxx/_git/RepositoryReferencedByTheSubmodule

Congratulant

Just to clarify something that wasn't obvious to me -- the git link is some kind of special format that refers to DevOps resources: git://YOUR_PROJECT_NAME/THE_SUBMODULE_REPO – Sagittal

It was complaining that couldn't check out some branch refs/heads/master and I had different branch name: main which wasn't default. When setting that as default, this worked. Otherwise, you may have to use this syntax: git://project/RepositoryReferencedByTheSubmodule@refs/heads/main. More examples: Checking out a specific ref. – Landin

Answer 3

I was just about to add an explicit checkout for my submodule (as suggested by @Fábio Machado de Oliveira) above, but I took another look at the project pipeline options.

My submodules are in the same project as the repositories that reference them, so the 3rd and 4th checkboxes aren't restricting access (they restrict to project-level scope, which is fine for me).

However, the last checkbox "Protect access to repositories in YAML pipelines", which "generates a job access token that is scoped to repositories that are explicitly referenced in the YAML pipeline" was the one that blocked access to implicitly referenced repositories in the same project.

While this might be a nice thing to have by default, with submodules, it means that I'd have to explicitly reference all sub-module repositories in all pipelines that have them. In my case the submodules are defined within the same project, so a project-scoped token is more than secure enough for me.

Rabbin