Using relative url for git submodule needs credentials

Question

The Team Services documentation (https://www.visualstudio.com/en-us/docs/build/define/repository#what-kinds-of-submodules-can-i-check-out) points out that I can do a $ git add submodule if

• it is an immediate submodule
• Unauthenticated (n/a)
• Authenticated
  • Contained in the same team project
  • Added by using a relative url from main repository

They give an example:

git submodule add /../../submodule.git mymodule

If I reference a git repo in the same project like

git submodule add ./../other-repo mymodule

It resolves the right repo but wants me to provide credentials. The build fails with the following message:

Cloning into 'mymodule'...
fatal: could not read Username for 'https://xxx.visualstudio.com': Invalid argument

Providing the full URL with credentials (https://user:[email protected]/...) works but is IMO a bad solution.

The documentation suggests that this should work with a relative url and without credentials. Am I wrong?

Edit 1:

Running with system.debug: true

Entering OnPrepareEnvironment
Primary repository: xxx
Calculating build folder hash key.
Loading tracking config if exists: C:\a\SourceRootMapping\07a8b96d-d805-4646-83d3-e7b2fbe394c2\18\SourceFolder.json
Creating new tracking config.
Loading top-level tracking config if exists: C:\a\SourceRootMapping\Mappings.json
Writing config to file: C:\a\SourceRootMapping\Mappings.json
Writing config to file: C:\a\SourceRootMapping\07a8b96d-d805-4646-83d3-e7b2fbe394c2\18\SourceFolder.json
Checking if artifacts directory exists: C:\a\1\a
Creating artifacts directory.
Checking if test results directory exists: C:\a\1\TestResults
Creating test results directory.
Creating binaries directory.
Setting local variables.
Create the initial timeline records for the tasks
Preparing repositories
repo clean = False
Found 3 endpoints to consider
Found 1 repositories to sync
Starting: Get sources
build.fetchtags=false
Entering GitSourceProvider.PrepareRepositoryAsync
Repository type=TfsGit
localPath=C:\a\1\s
clean=False
sourceBranch=refs/heads/r_080
sourceVersion=26d5a7a6e6ed47c8f12ee5dc5b376d6731b7863a
Syncing repository: xxx (Git)
repository url=https://xxx.visualstudio.com/_git/xxx
checkoutSubmodules=False
Starting clone
Checking out 26d5a7a6e6ed47c8f12ee5dc5b376d6731b7863a to C:\a\1\s
Checked out branch refs/heads/r_080 for repository xxx at commit 26d5a7a6e6ed47c8f12ee5dc5b376d6731b7863a
Leaving GitSourceProvider.PrepareRepositoryAsync
Leaving OnPrepareEnvironment
Running tasks
Starting task: Run git
##[warning]File name doesn't indicate a full path to a executable file.
Executing the following command-line. (workingFolder = C:\a\1\s)
git submodule add ./../other_repo mymodule
Error message highlight pattern: 
Warning message highlight pattern: 
C:\Windows\system32\cmd.exe /c "git submodule add ./../other_repo mymodule"
Cloning into 'mymodule'...
Fatal: InvalidOperationException encountered.
bash: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://xxx.visualstudio.com': Invalid argument
fatal: clone of 'https://xxx.visualstudio.com/_git/other_repo' into submodule path 'mymodule' failed
Finishing task: CmdLine
##[error]System.Exception: Task CmdLine failed. This caused the job to fail. Look at the logs for the task for more details.
##[error]   at Microsoft.TeamFoundation.DistributedTask.Worker.JobRunner.Run(IJobContext jobContext, IJobRequest job, IJobExtension jobExtension, CancellationTokenSource tokenSource)
Entering OnFinalizeJob
Leaving OnFinalizeJob

Edit 2:

I cannot answer this. The documentation is wrong/unclear, I went with the option to check out submodules like @eddie-msft said - but watch out: VSTS git build fails with git submodule

Angelika

I tried with you command to add a Git Submodule and queue a build but didn't see any issue. Can you queue the build with "system.debug" variable set to "true" and then share the entire logs? – Gaullist

@Eddie-MSFT added as Edit – Angelika

According to the logs, you are adding the submodule repo during a build process via "Command Line" build tasks. Can you try adding the submodule on your local machine and then pushing the changes? Please also try with "git submodule add ../other-repo mymodule" command. – Gaullist

I added it locally and tried to "git submodule update --init" in a build-step but that did just nothing. Using ../other-repo mymodule fails with the same error message. – Angelika

You don't need to get it in build step namually, just check "checkout submodule" option in the build definition repository settings. – Gaullist

Well, #38753129 – Angelika

It's a different behavior, in this case, the command line task does not have the credential information for authentication, you need to do this via "checkout submodules" option. And I added another point you need to pay attention if you use relative path. – Gaullist

See my related answer here: https://mcmap.net/q/1164857/-git-submodule-update-on-pipelines-with-hosted-agent, about reusing the main repo credentials for submodules in the same organization. – Cadet

Answer 1

I understand from the Submodules - Azure Repos that there are issues when:

• Submodule repo requires authentication.
• You have enabled Limit job authorization scope to referenced Azure DevOps repositories, which is enabled by default.

If you restricted the job access token as explained in the section above, then you won't be able to do this.

I have only tried nr 1 myself and it works for us.

Workaround 1: For submodule repo in DevOps

• Disable Limit job authorization scope to referenced Azure DevOps repositories in project settings, which grants the pipeline access to all DevOps repositories.
• Use relative paths instead of https:// in the submodule URL, such as ../MySubmoduleRepo or ../../../OtherProject/_git/MySubmoduleRepo.

Workaround 2: If different credentials are needed

• Use a script to checkout with a personal access token (PAT)

Workaround 3: Custom script to reuse access token for submodule sync

https://mcmap.net/q/1164857/-git-submodule-update-on-pipelines-with-hosted-agent

steps:
- checkout: self
  submodules: false
  persistCredentials : true

- powershell: |
    $header = "AUTHORIZATION: bearer $(System.AccessToken)"
    git -c http.extraheader="$header" submodule sync
    git -c http.extraheader="$header" submodule update --init --force --depth=1

Morven

Using relative path fixed it for my azure pipeline! Thank you – Berwickupontweed

Workaround 3 worked for me instantly, thanks! – Mystery

Answer 2

Please also check the path of your two repositories. If you have only two repositories in your project, the path of the first created repo A would be:

https://xxxx.visualstudio.com/_git/A

And the path of the second created repo B would be:

https://xxxx.visualstudio.com/A/_git/B

In this scenario, you need to use a relative path like following:

../../A/_git/B

Gaullist