Azure DevOps multistage pipeline YAML: how to checkout multiple repos?

Question

My Azure DevOps pipeline uses yaml templates from two different repos, which is configured as following.

1. There is an application repository which has deployable application and one yaml file - the pipeline's "root" template
2. Templates Repository. Root template calls other templates and stages from the Templates repository. Then templates from this repository call other templates and scripts (from the same repository)

Templates repository is referenced as a resource in the root template. I didn't find a way to checkout the templates repo just once and then use templates and scripts across all pipeline stages. For now I have to clone the templates repo manually in each stage where I need to use additional templates or scripts. At the end of each stage Azure Devops clears the cloned repo. Is there a simple way to checkout templates repo just once, or somehow else reference its resources from sub-stages?

Bamby

Answer 1

I'm not sure since you didn't show your YAML file, but did you use checkout step:

resources:
  repositories:
    - repository: devops
      type: github
      name: kmadof/devops-templates
      endpoint: kmadof

steps:
- checkout: self
- checkout: devops
- script: |
    echo $(Build.SourcesDirectory)
    ls $(Build.SourcesDirectory) *
- template: templates/template.yaml@devops
  parameters:
    repo: devops-templates

Above script checout two repos. In devops-templates I have template which is used in main build yaml file (located in self repo).

Please take a look also here

EDIT

I work a bit with this and tried few things. Let me describe first relation between files:

• build.yaml (main repo)
  • templates/start.yml (template repo - template with stages)
    • job one - templates/process.yaml (template repo)
      • steps - templates/another-template.yaml (template repo)
    • job two - steps directly defined in start.yaml

And you don't have to actually checkout template repo because at running all templates are fecthed and build plan is created. You only need to checkout template repo if you are going to run some scripts (for instance powershell scripts). Here you have my yaml files:

build.yaml

resources:
  repositories:
    - repository: devops
      type: github
      name: kmadof/devops-templates
      endpoint: kmadof

stages:
- template: templates/start.yaml@devops
  parameters:
    repo: devops-templates
    buildSteps:
      - checkout: self
      - checkout: devops
      - bash: echo Test #Passes
        displayName: succeed
      - bash: echo "Test"
        displayName: succeed

start.yaml

# File: start.yml
parameters:
- name: repo  # defaults for any parameters that aren't specified
  default: ''
- name: buildSteps # the name of the parameter is buildSteps
  type: stepList # data type is StepList
  default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
  pool: Hosted VS2017
  jobs:
  - template: process.yaml
    parameters:
      pool:   # this parameter is called `pool`
        vmImage: ubuntu-latest  # and it's a mapping rather than a string
  - job: secure_buildjob
    steps:
    - script: echo This happens before code 
      displayName: 'Base: Pre-build'
    - script: echo Building
      displayName: 'Base: Build'

    - ${{ each step in parameters.buildSteps }}:
      - ${{ each pair in step }}:
          ${{ pair.key }}: ${{ pair.value }}     

    - script: echo This happens after code
      displayName: 'Base: Signing'

process.yaml

parameters:
- name: 'pool'
  type: object
  default: {}

jobs:
- job: build
  pool: ${{ parameters.pool }}
  steps:
  - template: another-template.yaml
    parameters:
      repo: devops-templates

another-template.yaml

parameters:
- name: repo  # defaults for any parameters that aren't specified
  default: ''

steps:
  - pwsh: Write-Host 'Hello form another template'

Please take a look here:

Build job uses template from devops-template repo but I don't checkout repo in this job.

You may wonder why we can't have one checkout per build. And this is because each job can run a different agent.

Here you have few links:

• build log
• repo with main file
• repo with templates

Last remark, you really need to checkout repo with templates when you template calling a file from that repo. For instance:

steps:
  - task: PowerShell@2
    inputs:
      filePath: /scripts/myscript.ps1

Sponge

Hi Krzysztof, thanks for reply. with that approach I'm getting error Encountered error(s) while parsing pipeline YAML: ----- Unexpected value 'stages' My yaml is something like this: parameters: param1: '' stages: - stage: stage1 – Bamby

You probably copy pasted this and mixed steps with stages. Please check my edited answer. – Sponge

Thank you, I missed the possibility of passing steps into templates. Now it works – Bamby

Cool. I am happy to hear that. Can you consider upvoting my answer if it were useful for you? – Sponge

Hey @KrzysztofMadej, I have similar issue and what is the endpoint here? Which repo it should be created in? self or the other which I try to checkout? – Egyptian

Thank you for the time you put into this!!! I got my pipeline working!!! :) – Janus