Is it possible to use a variable in the ref property of resources:repository for Azure DevOps YAML?

Question

I have two AzureDevOps Git branches:

master
feature/mybranch

I have a multi-stage build pipeline defined in yaml, where some of the steps are templated into separate .yml files.

In my outer azure-pipelines.yml I reference a repository where my template .yml's live:

resources:
  repositories:
    - repository: templates
      type: git
      name: MyProject/MyRepo

when I'm building in the 'master' branch everything is good as by default the repository will look in refs/heads/master.

when I'm working in the feature branch and I want to test experimental changes to my template .yml files, I don't want it to fetch them from the master branch, I want it to use the files from the branch I am working in.

The following works and allows me to do this:

resources:
  repositories:
    - repository: templates
      type: git
      name: MyProject/MyRepo
      ref: refs/heads/feature/mybranch

However, when I merge this back to master, I obviously don't want 'ref:' still pointing at the feature branch, so I'd like to generate the value of 'ref:' dynamically with a variable.

I've tried using ref: $(Build.SourceBranch) where $(Build.SourceBranch) should expand to 'refs/heads/feature/mybranch'

But it doesn't work. Error:

62638: "/azure-pipelines.yml: Could not get the latest source version for repository MySolution hosted on Azure Repos using ref refs/heads/$(Build.SourceBranch)."

Paramagnetism

What do you mean by the "current" branch? If you queue a build, you can choose the branch. If a build is triggered by a CI trigger, it will automatically use the branch that triggered CI. There's nothing in YAML you have to specify for this behavior. – Anode

Daniel Mann - I've updated the question to hopefully make it clearer what I'm trying to achieve – Paramagnetism

Does your issue solved now? Feel free tot let us know the latest status. – Paramour

Answer 1

Instead of referencing the repo in resources, use inline checkout as described here

https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checking-out-a-specific-ref

- checkout: git://MyProject/MyRepo@features/tools

And this yaml element allows use of template expressions using variables, parameters e.g.

- checkout: git://${{ variables.repoName}}@${{ variables.branchRef }}

OR

 - checkout: git://${{ parameters.repoName}}@${{ parameters.branchRef }}

And you can change that dynamically

Or the other alternative is use script task as below

- script: |
      # note checkout: git://$(System.TeamProject)/${{ parameters.repoName }}@${{ parameters.repoRef }} this does not work if this task is run multiple times in same pipeline
      # see here for more details :https://developercommunity.visualstudio.com/t/checkout-fails-with-an-error-occurred-while-loadin/1005991#T-N1270459
      repoDir=$(Agent.BuildDirectory)/${{ parameters.repoName }}
      /bin/rm -rf $repoDir
      url_with_token=$(echo $(System.CollectionUri) | sed -e "s/https\:\/\//https\:\/\/$(System.AccessToken)\@/g")
      git clone $url_with_token/$(System.TeamProject)/_git/${{ parameters.repoName }} $repoDir
      cd $repoDir
      git fetch origin '${{ parameters.repoRef }}':'localBranch'
      git checkout localBranch
    name: clone_script
    displayName: Checkout using script ${{ parameters.repoName }}@${{ parameters.repoRef }}
    

I have added above as a template and its usage in a gist, to make it re-usable easily.

Hope that helps

https://gist.github.com/scorpionlion/1773d08b62ca5875cc2fd6dcdd0394d2

Photocell

This is fantastic. I needed to be able to load different repos with the same branch name as my main repo and had not been able to find a solution to allow that before this one. Hopefully they will get the ability to use it directly implemented soon, but until then, this should get the job done. – Mover

be aware of one issue developercommunity.visualstudio.com/t/… – Photocell

added alternative checkout as well – Photocell

if I'm reading that correctly, it looks like the problem is the runtime interpretation of the repo URI. In that case, using the "compile" time syntax ${{ }} may work. Just need to make sure there are no run-time evaluations in it's contents. At least that's what Andy Li's post would make me expect. – Mover

Exactly ..do not use macro variables like $() ...rather only template expression ${{}} works, which gets expanded before queuing a pipeline – Photocell

The System.CollectionUri is equl for example: dev.azure.com/fabrikamfiber, so you cannot use it in this form together with System.AccessToken. See learn.microsoft.com/en-us/azure/devops/pipelines/build/… – Scotch

corrected the url with token – Photocell

@Photocell May I ask why did you use macro variables like $() in your sample checkout steps ? Because macro variables should not work in checkout steps as far as I see – Scientist

Kagan - good spot - I the $() macro never worked, but gave empty values. I have updated the answer! – Photocell

Hi @Photocell Is approach - checkout: git://${{ variables.repoName}}@${{ variables.branchRef }} is working ? Because I am always getting error as "The String must have at least one character. Parameter name:repositoryName" while trying to run my job. – Packing

@Packing I think you are right. Variables do not work! Parametes work. – Eduard

this only works if you have defined the variable in yaml and variable is available at que time, if you want to use runtime variables, then I suggest using the gist – Photocell

@Photocell What do you mean with "using the gist"? You want that I show my code in github gist? – Eduard

What is mean is use the template from this [gist ](gist.github.com/scorpionlion/1773d08b62ca5875cc2fd6dcdd0394d2) – Photocell

Answer 2

According to https://learn.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-212-update#template-expressions-in-repository-resource-definition you need to use Template Expressions. I had cases where this worked (using variables), and in other pipelines, where using variables did not work

resources:
  repositories:
    - repository: templates
      type: git
      name: MyApp/MyconApp
      ref: $(Build.SourceBranchName)

In cases where it did not work, changing to Template Expression made it work.

resources:
  repositories:
    - repository: templates
      type: git
      name: MyApp/MyconApp
      ref: ${{ variables['Build.SourceBranch'] }}

Biophysics

Answer 3

Is it possible to use a variable in the ref property of resources:repository for Azure DevOps YAML?

For this question, the answer is Yes, it's possible.

About why you receive that error message, just is the variable($(Build.SourceBranch)) you used is incorrect. You should use $(Build.SourceBranchName).

As normal, for ref, we should input master or any other feature branches. Such as

ref: refs/heads/master

This may make you thought that this is same with the value of $(Build.SourceBranch). It looks same, I know, but different. In fact, for server, it will read the exactly branch name not the branch path, which we can clearly figure out with the classic editor type:

According with classic editor type, we can know here we should input the exactly branch name.

So, as the Predefined variables defined, the value of $(Build.SourceBranch) is the branch path, but for $(Build.SourceBranchName), it's represent a exactly branch name.

So, if you want to execute successfully, you need to use : $(Build.SourceBranchName). And it's worked on my side.

Hope this also can help you stay away from the error message.

Edit:

The complete script which is worked for me is:

resources:
  repositories:
    - repository: templates
      type: git
      name: MyApp/MyconApp
      ref: $(Build.SourceBranchName)

Paramour

Many thanks for looking into this for me. Unfortunately, your proposed solution didn't work for me: 63049: "/DevOps/DevOps.Pipeline/azure-pipelines.yml: Could not get the latest source version for repository MyRepo hosted on Azure Repos using ref refs/heads/$(Build.SourceBranchName)." – Paramagnetism

That's wired. What's your script like? For me, the worked script is ref:$(Build.SourceBranchName). I update my answer, please check it. – Paramour

@MerlinLiang-MSFT - I'd be interested to know if this approach is still working for you. I am using the approach you outlined, yet I am encountering the same errors that MarkdotH described. – Intense

Seems as though it isn't yet supported, but is being reviewed developercommunity.visualstudio.com/idea/816606/… – Anastigmat

Answer 4

The azure docs state

Variables can't be used to define a repository in a YAML statement.

So that seems to place some limitations on what you can do here. Perhaps there is a workaround that still allows you to do what you want.

Exclamation