How to get only changed files using Azure devops pipelines

Question

I have folder structure this way in source code. f1 f2 f3 f4

I have added gitcopy diff task in my pipeline which lists and copies files which are modified to a destination folder. Now, I want to have a conditional loop as powershell script to only zip those folders which have modified files with a particular names for example if files from f1 are modified..I want particular steps to be performed and so on.. How can I do it as a loop? edit: I have written my pipeline in this way. But its failing in publish step with errors as listed.

yaml: trigger:

none

pool:
  vmImage: 'windows-latest'

variables:
  FR1PHPPRDAPP1VFlag: false
  FR1PHPPRDAPP4VFlag: false
  FR1PHPPRDAPP5VFlag: false
  FR1PHPPRDSRE1VFlag: false
  FR1PHPPRDAPP7VFlag: false
  stages:
  -stage: Zipping modified folders
steps:

- powershell: |

      ## get the changed files
      $files=$(git diff HEAD HEAD~ --name-only)
      $temp=$files -split ' '
      $count=$temp.Length
      echo "Total changed $count files"
      For ($i=0; $i -lt $temp.Length; $i++)
        {
          
          $name=$temp[$i]
          echo "this is $name file"
          if ($name -like 'FR1PHPPRDAPP1V/*') 
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
           
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP1V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP1V.zip
          
          ##set the flag variable FR1PHPPRDAPP1VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP1VFlag]true"
          }
          if ($name -like 'FR1PHPPRDAPP4V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP4V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP4V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP4V.zip
          ##set the flag variable FR1PHPPRDAPP4VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP4VFlag]True"
          }
           if ($name -like 'FR1PHPPRDAPP5V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP5V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP5V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP5V.zip
          ##set the flag variable FR1PHPPRDAPP5VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP5VFlag]True"
          }
            if ($name -like 'FR1PHPPRDSRE1V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDSRE1V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDSRE1V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/SRE1V.zip
          ##set the flag variable FR1PHPPRDSRE1VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDSRE1VFlag]True"
          }
            if ($name -like 'FR1PHPPRDAPP7V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP7V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP7V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP7V.zip
          ##set the flag variable FR1PHPPRDAPP7VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP7VFlag]True"
          }
        }
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/Output'
    ArtifactName: 'scripts-f2p'
    publishLocation: 'Container'
  condition: and(succeeded(), or(eq(variables.FR1PHPPRDAPP1VFlag, true),eq(variables.FR1PHPPRDAPP4VFlag, true),eq(variables.FR1PHPPRDAPP5VFlag, true),eq(variables.FR1PHPPRDSRE1VFlag, true),eq(variables.FR1PHPPRDAPP7VFlag, true)))

Electrum

Answer 1

You can directly run below git commands in the powershell task to check the changed files. It is much easier than Rest api.

git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)

When you get the changed files, you can use the zip the changed folders directly in the powershell task using Compress-Archive command: See below example:

Compress-Archive -Path C:\f1 -DestinationPath f1.zip

If you want some particular steps to be performed based on the the changed folders. You can define the flag variables and using the logging commands in the powershell scripts to set the flags to true. And then use the condtions for the following steps.

See below full example scripts:

##set flag variables to indicate if the folder is changed.

variables:
  f1Flag: false
  f2Flag: false
  f3Flag: false
  
steps:

- powershell: |
      ## get the changed files
      $files=$(git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion))
      $temp=$files -split ' '
      $count=$temp.Length
      echo "Total changed $count files"
     
      For ($i=0; $i -lt $temp.Length; $i++)
      {
        $name=$temp[$i]
        echo "this is $name file"
        if ($name -like 'f1/*')  #if f1 is a subfolder under a folder use "- like '*/f1/*'"
        { 
          ##achive folder f1 if it is changed.
          ##Compress-Archive -Path $(system.defaultworkingdirectory)/f1 -DestinationPath $(Build.ArtifactStagingDirectory)/f1.zip
          
          ##set the flag variable f1Flag to true
          Write-Host "##vso[task.setvariable variable=f2Flag]true"
        }
        if ($name -like 'f2/*')
        {
          ##achive folder f2 if it is changed.
          ##Compress-Archive -Path $(system.defaultworkingdirectory)/f2 -DestinationPath $(Build.ArtifactStagingDirectory)/f2.zip
          ##set the flag variable f2Flag to true
          Write-Host "##vso[task.setvariable variable=f2Flag]True"
        }
      }
      ## create a temp folder to hold the changed files
      New-Item -ItemType directory -Path $(system.defaultworkingdirectory)\temp

      foreach($file in $temp){
        if(Test-Path -path $file){
        Copy-Item -Path $file -Destination $(system.defaultworkingdirectory)\temp
        }
      }
      ## zip the temp folder which only have the changed files
      Compress-Archive -Path $(system.defaultworkingdirectory)\temp\* -DestinationPath $(Build.ArtifactStagingDirectory)\changedfiles.zip

Then you can use the condition for some particular steps just as Krzysztof mentioned

condition: and(succeeded(), or(eq(variables.f1Flag, true),eq(variables.f2Flag, true),eq(variables.f3Flag, true)))

See the answer to this thread for more information.

Update:

steps:

- powershell: |
     #get the changed files
     ....

        
- task: PublishBuildArtifacts@1
  inputs:
     PathtoPublish: '$(Build.ArtifactStagingDirectory)/Output'
     ArtifactName: 'drop'
     publishLocation: 'Container'
  condtion: and(succeeded(), or(eq(variables.f1Flag, true),eq(variables.f2Flag, true),eq(variables.f3Flag, true)))
   

Update 2:

The default checkout is now a shallow clone of depth 1 preventing the git diff to work. Checking out with fetchDepth: 0 solves that problem as mentioned in the comments. This may increase the time required for the checkout step - especially on larger repos.

Cory

Hi, could u check my edit? I have made changes to the pipeline.Now its failing in the condition statement – Electrum

@Electrum You can must use stage under stages section. you can use condtions for steps. no need to use stage. See above update – Cory

@Electrum How was it going? Did it work out? – Cory

@ lEVI, thankyou..could u pls tell me how can I add multiple conditions here? I want the publish task to run when either f1 or f2 or f3 flag turns true.. – Electrum

@Electrum You can use or expression. See above updated. Check here for more information. – Cory

@ievi, thankyou.. but when am using this pipeline along with the changes..all the folder contents are coming..I just want only the modified file to be coming in the artifact download...pls check my whole pipeline in the edit – Electrum

@priya. You can just copy the modified files to a folder and then zip it. See above updated scripts in powershell task. – Cory

@Levi..thankyou very much for the help..just realized this solution works well if I have modified/added files..but build is failing when I renamed the file name..could you suggest me what changes can we make to even zip the files which got renamed? – Electrum

@priya. you can use git diff-tree --no-commit-id commands. And check if the file exists using Test-Path -path. See above updated scripts. – Cory

@Levi..this helped very much..Thankyou Levi :) – Electrum

it looks like this only works if you do a single commit per push, otherwise the build.sourceversion is only one of the commits that were pushed. Has anyone figured out a way to handle a multi-commit push? – Eudora

it also appears that if the previous build fails, it only provides the commit id of the current attempt, not the previous attempts. i found a way to get a set of commits from a starting version, but i need to know the commit id of the previous succesful build. – Eudora

I made some edits to handle using this in a pull request and with file names with spaces. In case the edits don't get added: Add a variable isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')], if ("${{ variables.isMain }}" -eq "true") { $changedFiles=$(git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)) } else { $changedFiles=$(git diff --name-only origin/main) }, Foreach ($name in $changedFiles). – Obed

Oops it should be if ("$(isMain)" -eq "true") .... – Obed

@Eudora You may be able to use the (Cache)[learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/… task to store the built commit id [as a last step when the build succeeds] and then do the git diff against that on subsequent runs. – Stonewort

@LeviLu-MSFT is it possible to publish the changed files artifact to remote FTP server using above yml script? – Bekha

If, like me, you're confused why this doesn't work in the pipeline but works locally when you test it, it might be because the default clone behaviour changed in the pipeline setup; it now defaults to shallow clone, with a depth of 1, so therefore you need to explicitly override this in your checkout step to get the commit before HEAD: fetchDepth: 0 – Repertoire

For this to work properly, a build variable as Build.TargetVersion should be exposed by ADO, just like how Build.SourceVersion is. This is a miss by ADO, imo. This was it will work in all cases (single commit branch, multi commit branch, a feature branch merging into another feature branch). Without this, the target commit will need to be hard-coded to something as develop, master, main branch, and that means the pipeline template will have caveats. :shrug: I'm not sure if System.PullRequest.TargetBranch works. – Oarfish

I tried the above mentioned, but when i run the pipeline it errors out at PublishBuildArtifacts task with the below error: [error]Publishing build artifacts failed with an error: Not found PathtoPublish: D:\a\1\a\temp_notebooks – Rattly

follow-up to my above comment, i forgot to mention that I have the similar situation, just that i dont want to create a zip file, but the whole list of changed files. So I tried to use a copyfiles task and publishbuildartifacts tasks to do it, but when i run the pipeline it errors out at PublishBuildArtifacts with the above error. – Rattly

Answer 2

If you are creating a pipeline that checks files changed in an Azure Devops PR, you can create a Bash or Powershell task and use this command to get a list of the files changed:

Bash

git diff --name-only @~ @

Powershell

git diff --name-only HEAD~ HEAD

The reason this works is that for an Azure Repos Git PR, in the pipeline's local branch the git history contains one merge commit for all of the changes in the PR. It's just one merge commit regardless of how many commits were contained in the PR.

Personate

Thanks, works perfectly! Much easier than the other answers too! – Anaphylaxis

how do you use @~ @ in powershell? it's unrecognize! maybe with "@~ @"? – Discommodity

@ShoshanaTzi @ is also known as HEAD. So you could also use: git diff --name-only HEAD~ HEAD – Personate

Like the above answer, this also now requires you to set fetchDepth: 0 in the checkout step to get sufficient repo history for the comparison. – Repertoire

Answer 3

I managed to get changed files using the following powershell script using azure devops REST API:

# remove 'refs/head' that is prefixed to branch names
$sourceBranch = "$(System.PullRequest.SourceBranch)"
$sourceBranch = $sourceBranch.Replace("refs/heads/", "")
$targetBranch = "$(System.PullRequest.TargetBranch)"
$targetBranch = $targetBranch.Replace("refs/heads/", "")

# auth headers
$headers=@{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("azdo:$(System.AccessToken)")) }

# get PR diff changes from API
$response = (Invoke-WebRequest -Uri "https://{instance}[/{team-project}]/_apis/git/repositories/$(Build.Repository.ID)/diffs/commits?baseVersion=$targetBranch&targetVersion=$sourceBranch&api-version=5.1" -Method GET -Headers $headers  | ConvertFrom-Json)

# get path to changed files only, join them by spaces
$paths = $response.changes | Where-Object { -not $_.item.isFolder -or $_.item.isFolder -eq $false } | ForEach-Object { $_.item.path }

Bawdy

Getting git to behave was strangely not working for me, the api helped out a lot! – Horsemanship

Answer 4

If anyone is reading this and what happens is that the git-diff or any command does not work correctly (does not get diffs or only a single commit with all the repository content) to try the following: steps:

• checkout: self
  • clean: true
  • persistCredentials: true
  • fetchDepth: 0

FetchDepth is the key

Buffalo

Answer 5

There is nothing out of the box solution. But you can use REST API Commits - Get Changes call to check that:

GET https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/commits/be67f8871a4d2c75f13a51c1d3c30ac0d74d4ef4/changes?top=2&skip=10&api-version=5.0

{
  "changeCounts": {
    "Add": 456
  },
  "changes": [
    {
      "item": {
        "gitObjectType": "blob",
        "path": "/MyWebSite/MyWebSite/favicon.ico",
        "url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite/MyWebSite/favicon.ico?versionType=Commit"
      },
      "changeType": "add"
    },
    {
      "item": {
        "gitObjectType": "tree",
        "path": "/MyWebSite/MyWebSite/fonts",
        "isFolder": true,
        "url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite/MyWebSite/fonts?versionType=Commit"
      },
      "changeType": "add"
    }
  ]
}

If you use gitcopy diff task and you didn't flatten destination folder you should analyze you destination folder using command line - powershell/bashh - and based on this set variables.

$f1Changed = ((Get-ChildItem -Path .\destination\f1\ | Measure-Object).Count -gt 0)

Then you can analyze this response and check what was changed. Based on that you can set variables using logging command:

Write-Host "##vso[task.setvariable variable=f1changed;]$f1Changed"

And then use this variable in condition on your step here zip this particular folder.

 condition: and(succeeded(), eq(variables.f1changed, true))

Rakes

I am using git copy diff task for this purpose..now I have the changed files.But I want to perform zip operation on changed files which can be part of any of the folders..could u revisit my question please..I have edted it – Electrum

Please check me edit. I added there powershell script to check if there are files in destination/f1 folder and then used this in logging command. – Rakes

@krzystof Hi thanks for suggesting...could you check my edit please? I do not need the whole folder to be zipped..I jst need the changed files to get zipped in the destination folder..could u suggest any changes? – Electrum

Well in this case please just zip your destination folder when you copied your just modified files. I missed that part that you want to copy just modified files. – Rakes

I am not using copy diff task here..using powershell task to get the modified files through git diff command. check here https://mcmap.net/q/795922/-how-to-get-only-changed-files-using-azure-devops-pipelines/13460189 – Electrum

In this case since you know what was changed just copy this files to folder which you later zipped. – Rakes

Answer 6

git diff does not give expected results when used alongside the shallow checkout optimization. Shallow checkout may be applied by default on some versions of DevOps since 2022. Disabling shallow checkout with fetchDepth: 0 incurs a performance hit on larger repos. Calling the DevOps API to perform the diff works, but the script is lengthy.

We can use git fetch with --unshallow to get the current branch's history. The branch history provides context for git diff to yield the expected result without sacrificing performance as much as disabling shallow checkout.

- task: PowerShell@2
    displayName: 'Get a list of files changed by this branch'
    inputs:
      targetType: 'inline'
      script: |
        # Fetch full history of current branch for usage in git diff
        git fetch origin $(git rev-parse HEAD) --unshallow
        git fetch origin main

        # Get a list of files changed by this branch compared to origin/main
        $changedFiles = git diff origin/main...HEAD --name-only

Robb