Run Azure Pipelines job only when certain files are changed

Question

I have a repository that contains both a Dockerfile and other code. We want to run certain steps regardless of what changed, but only want to run the docker build job when **/Dockerfile is changed.

I've reviewed the Conditions documentation and the Expressions documentation but it's unclear to me how (if it's possible at all) how to combine these or otherwise achieve the desired outcome.

I realize it's possible in bash (e.g. git rev-list... and git diff --name-only <previous_commit>..HEAD | grep <pattern>) but this is a bit cumbersome and it still shows in Azure Pipelines that the job ran, it just short-circuited. Ideally it would show (appropriately) that the job was skipped all together.

I also realize that that the Docker portion and the code portion could be in separate repositories with separate build triggers, but would like to keep them together in the same repo if possible.

Biogeography

I'm not certain what you describe is possible. Perhaps if the docker build job depended on another job that succeeded or failed based on whether the Dockerfile had changed? But even if the file itself hasn't changed, its inputs may have, it's not clear why you want to skip the check that it can be built. – Tambour

@Tambour It's mostly to preserve build time – agree on the inputs, hadn't considered that. Leaning towards extracting the docker portion to another repo now. Thanks for the input. – Biogeography

You could create a job called Should run docker? and set a variable there, then use that variable on the job Building docker container, that way it's clear whether docker was built or not. – Vastah

Answer 1

Sorry but there's no trigger per job. Trigger is for pipeline scope.

According to your requirements, you may take this structure as a workaround:

jobs: 
  - job: OtherSteps
    steps:
    Your other steps in this job.
    - task: PowerShell@2
      inputs:
        targetType: 'inline'
        script: |
          $changedfiles = git diff ... (Choose right git command depending on your needs.)
          Write-Host $changedfiles
          If ($changedfiles.Contains("Dockerfile"))  {
            echo "##vso[task.setvariable variable=IfRunDockerBuild;isOutput=true]run"
          }
      name: 'DetermineIfRunNextJob'

  - job: DockerBuild
    dependsOn: OtherSteps
    condition: eq(dependencies.OtherSteps.outputs['DetermineIfRunNextJob.IfRunDockerBuild'],'run')
    steps:
    - script: echo Only run this job when IfRunDockerBuild=run instead of Null!

1.Assuming your have job1 and job2(docker build), you just need to add one PS task like above in the end of the job1. Then it outputs one variable which determines if we need to run job2 or skip it.

2.The Powershell task can run on Linux, macOS, or Windows.

2.The core of this workaround comes from this feature: Use the output variable from a job in a condition in a subsequent job.

Whitt

This is good. Was trying to avoid the git diff pieces I mentioned in the original question, but that was assuming it was in a single step. Seems logical to extract the logic into a predecessor step. – Biogeography

Remember: if your build is configured to do batching, you may be building more than 1 commit. In which case you may need to use the REST API to get the last successful commit hash to do the diff against. Having a separate build definition isn't such a bad idea ;). – Vastah

We use semantic-release which adds a semver tag on successful build. I'm looking back through the tree to the latest tag and running the diff against that. But yes, still strongly considering just extracting these bits. – Biogeography

Answer 2

Although the question is old, I had the same problem and I think I have a good solution. The challenge is to ensure that the solution works even if multiple commits are pushed simultaneously, or if a build fails (and therefore doesn't deploy) or on PR merges where deployments only happen on the main branch.

I've described my solution in full in this gist: https://gist.github.com/Myrddraal/f5a84cf3242e3b3804fa727005ed2786

It takes advantage of the pipelines API, which can provide a list of all commits since the last successful pipeline execution. This ensures that it works even when multiple commits are pushed at once, or when the build with the infrastructure change failed. The pipelines API does the hard work of working out which commits need checking.

The logic is in this powershell:

[CmdletBinding()]
param (
  $authorisation,
  $pathFilter,
  $collectionUri,
  $project,
  $buildId
)

$changesUrl = "$collectionUri/$project/_apis/build/builds/$buildId/changes?api-version=6.0"
$changesResponse = Invoke-RestMethod -Uri $changesUrl -Headers @{Authorization = $authorisation } -Method Get
$commits = @($changesResponse.value | ForEach-Object { $_.id })

Write-Host "##vso[task.setvariable variable=filesUpdated;isOutput=true]False"
Write-Host "Checking $($commits.Length) commits for changes matching path $pathFilter"
for ($j = 0; $j -lt $commits.Length; $j++) {
  Write-Host "Checking commit: $($commits[$j]) with its parent"
  $files = $(git diff "$($commits[$j])~" $commits[$j] --name-only)
  Write-Host $files
  if ($files -like "*$pathFilter/*") {
    Write-Host "Found file matching path filter in commit $($commits[$j])"
    Write-Host "##vso[task.setvariable variable=filesUpdated;isOutput=true]True"
    break
  }
}

Invoke it with the following YAML (in a build job after pulling the repository):

  - task: PowerShell@2
    inputs:
      filePath: "azure-pipelines/Test-ChangesMadeInPath.ps1"
      arguments: >-
        -authorisation "Bearer $(system.accesstoken)" 
        -pathFilter "azure-pipelines/deployment" 
        -buildId $(Build.BuildId)'
        -collectionUri $(System.CollectionUri)
        -project $(System.TeamProject)
    name: DetermineChangesMadeInDeploymentFolder
    env:
      SYSTEM_ACCESSTOKEN: $(system.accesstoken)

Then add the following condition to your deployment job:

  - deployment: DeployInfrastructure
    condition: eq(stageDependencies.Build.BuildJob.outputs['DetermineChangesMadeInDeploymentFolder.filesUpdated'], 'True')
    displayName: Deploy infrastructure
    environment: "prod"
    strategy:
      runOnce:
        deploy:
          steps:
            - template: deployment/YAML/deploy-infrastructure.yml
              parameters:
                environment: $(Environment.Name)

Feodore

Answer 3

You can define a paths/include as well as paths/exclude filter on each trigger. I couldn't find this on the regular docs site, but the YAML repo clearly explains it:

Example:

trigger:
  batch: true
  branches:
    include:
    - features/*
    exclude:
    - features/experimental/*
  paths:
    include:
    - **/Dockerfile

PS: Not entirely sure whether wildcards are supported and what syntax to use for them.

Vastah

Forgive my ignorance (I'm fairly new to Azure Pipelines), but is there a way to attach a trigger per job? It appears from the docs that triggers are associated with the whole pipeline, not jobs or stages within the pipeline. – Biogeography

@Biogeography It's by design that we can't set trigger per job, check if my idea helps :) – Whitt

Answer 4

Note: DaveF's answer works well, but you now have to add specify the preceding checkout step as:

steps:
- checkout: self
  fetchDepth: 0

See ambiguous argument 'HEAD^' in Azure DevOps pipeline for further info

Bald