Azure Devops pipeline --> Git clone with service Principal

Question

I previously was using a PAT token to authenticate with Azure DevOPS GIT in order to download submodules. Now I am trying to authenticate with a Service Principal since it is newly supported by Microsoft.

Given documentation and here, I came up with the following code which I should expect to work however the git clone step is not working correctly. The first half of the script seems to work because I am able to successfully retrieve a token. The error I am getting is fatal: could not read Password for 'https://[email protected]': terminal prompts disabled

 - task: Bash@3
  displayName: Fetch Submodules
  inputs:
    targetType: 'inline'
    script: |
      creds=$(curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=$client_id&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&client_secret=$client_secret&grant_type=client_credentials" https://login.microsoftonline.com/$tenant_id/oauth2/v2.0/token)
      creds=$(echo "$creds" | jq -r .access_token)
      git -c http.https://dev.azure.com/conso/DevOps/_git/terraform.extraheader="AUTHORIZATION: Bearer $creds" submodule update --init --recursive
  env:
    client_id: xxx
    client_secret: "xxxx"
    tenant_id: xxx

The service principal has been added to Azure Devops User and has also been give read access to the repository I am cloning

Kittle

Answer 1

So after a lot of trial and error, i've retweaked the bash script so that it looks something like this.

The code below works within an Azure devops pipeline, can clone git repos and can also clone submodules.

          resource="499b84ac-1321-427f-aa17-267ca6975798"

          # Obtain the Azure AD authentication endpoint
          auth_url="https://login.microsoftonline.com/$tenant_id/oauth2/token"

          # Construct the data payload for the token request
          data="grant_type=client_credentials&client_id=$client_id&client_secret=$client_secret&resource=$resource"

          # Send a POST request to the authentication endpoint and capture the response
          access_token=$(curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "$data" "$auth_url" | jq -r .access_token)

          # Pull git repository
          git clone https://[email protected]/organisation/project/_git/myGitRepo myCustomFolder

          # Pull submodules
          git -c http.https://dev.azure.com/organisation/project/_git/myGitRepo.extraheader="AUTHORIZATION: Bearer $access_token" submodule update --init --recursive

Kittle

Answer 2

There are a few ways to achieve what you are trying to do. The most recent and comprehensive answer is in my SO answer on a related topic here.

You have three options to perform the task you want:

1. Follow the link above to add a Service Principal to the Azure Active Directory which is linked to the Azure DevOps Organisation, generate a secret for it, and add it to the repositories in question with Read access on the specific repository.

2. If the projects are part of the same organisation, you can add the Project Collection Build Service from your project to the source project's repository with Read access. You can then use a combination of a repository resource (more info here) and the standard checkout step to clone the target resource. If you do this, you can set the submodules property of the checkout step to true and it will handle the submodule initialisation for you. Note that your Project Collection Build Service user must have read on the submodule repositories also.

   If you want to run commands in this folder afterwards, you can also set persistCredentials to true which will mean you can then run other commands (like submodule init or push) inside that repository once it has been cloned - assuming your Project Collection Build Service has permissions.

3. If the projects are part of the same organisation, you can try manually running the clone step using "classic" git commands inside a bash shell. You can do this by exposing the System.AccessToken which is documented in more detail in the microsoft documentation on how to use and access it. Note that as the System.AccessToken variable is sensitive, you must pass it in via environment variables to your script (it will not be replaced directly in inline scripts like a usual variable). Once you have this, you can simply use something resembling the below in your clone script:

   git clone https://[ACCESS_TOKEN]@dev.azure.com/[YOUR_ORG]/[YOUR_PROJECT]/_git/[YOUR_REPOSITORY]

   If you do this, you'll be able to use all the standard commands such as submodule init etc.

Note that Option 3 above is actually identical to Option 2, because that's the commands that Azure DevOps will run for you as part of the checkout step - you just have more access to the commands used to do it.

Personally, if you can control the Azure Active Directory that is backing the Azure DevOps Organisation in question, Option 1 is the cleanest. You are completely disconnected in your consuming pipeline from the source. If both projects are part of the same Organisation, Option 2 is cleaner as there's no secrets to pass around - you just use the Build Service identity and it's all kept nice and tidy (and secure) in Microsoft's stack.

Edit For those visiting who want to do this purely through cURL, this blog post has a brilliant overview.

Repeating the key elements here in case that link dies,

curl https://login.microsoftonline.com/<tenant-ID>/oauth2/token 
-H "Content-Type: application/x-www-form-urlencoded" 
--data
"
    grant_type=client_credentials&
    client_id=<Client ID>&
    client_secret=<Client Secret>&
    resource=499b84ac-1321-427f-aa17-267ca6975798
"

Interestingly that post goes on to remark that you can use the Tenant Name rather than Tenant ID, though I have not tested this.

Sinfonia

How would I go about doing this if I want to use curl to retrieve the bearer token? This is currently biggest. In my case all the the git repos are located in totally different Azure Devops clients/ organisations, hence I do not have the option of using the inbuild tooling and it needs to be using client_id, client_secret and tenant_id. – Kittle

@Kittle if you absolutely cannot use the Azure CLI - which I strongly, strongly recommend you do if you can - then you can interact directly with the Azure API using web requests. This is documented excellently in this blog post (jiasli.github.io/azure-notes/aad/…), and I will update my answer to include a link to this. – Sinfonia

Thanks for this, your help was very valuable. However just a note that the resource is incorrect in your post and has to be resource=499b84ac-1321-427f-aa17-267ca6975798 – Kittle

@Kittle thanks for the feedback, I have corrected the answer accordingly. Please go ahead and mark as answer if you've solved your problem! – Sinfonia

Answer 3

Note : When creating custom groups in Azure DevOps to add the service principal to you need that group to have the "View project level information" permission(part of Readers group usually ) or else it's going to return errors . Regular Azure DevOps groups won't have that issue .

Erase