How to trigger a build only if changes happen on particular set of files

Question

How do I tell Jenkins/Hudson to trigger a build only for changes on a particular project in my Git tree?

Adalai

Answer 1

The Git plugin has an option (excluded region) to use regexes to determine whether to skip building based on whether files in the commit match the excluded region regex.

Unfortunately, the stock Git plugin does not have a "included region" feature at this time (1.15). However, someone posted patches on GitHub that work on Jenkins and Hudson that implement the feature you want.

It is a little work to build, but it works as advertised and has been extremely useful since one of my Git trees has multiple independent projects.

https://github.com/jenkinsci/git-plugin/pull/49

Update: The Git plugin (1.16) now has the 'included' region feature.

Retarded

1.1.16 is the correct version number for the included feature. (there is no 1.16) – Kamp

I can't make it works, I have a repositorie with multiple modules (domain, common, api, desktop_app,...) I want trigger a build for desktop_app for example, I put on the "included regions" production_app/*, I tried several combinations like ./desktop_app even absolute path. AndI always got Ignored commit c6e2b1dca0d1885: No paths matched included region whitelist. Any clue? More details here: #47439542 – Wilton

Bitbucket users: I might be wrong, but this approach doesn't seem to work. I'm posting a workaway solution. – Unscramble

Answer 2

If you are using a declarative syntax of Jenkinsfile to describe your building pipeline, you can use changeset condition to limit stage execution only to the case when specific files are changed. This is now a standard feature of Jenkins and does not require any additional configruation/software.

stages {
    stage('Nginx') {
        when { changeset "nginx/*"}
        steps {
            sh "make build-nginx"
            sh "make start-nginx"
        }
    }
}

You can combine multiple conditions using anyOf or allOf keywords for OR or AND behaviour accordingly:

when {
    anyOf {
        changeset "nginx/**"
        changeset "fluent-bit/**"
    }
}
steps {
    sh "make build-nginx"
    sh "make start-nginx"
}

Dismissal

Keep in mind it does not work for some cases. Refer to issues.jenkins-ci.org/browse/JENKINS-26354 for more details. – Dehiscent

Also keep in mind that when you use this feature in a multi module build (e.g. Maven reactor), it will not resolve and build dependencies between those modules. To achieve that, you would rather need a build plugin that understands you build and dependencies. – Eclogue

If you want to have the changeset generated between two specific branches (e.g. if you are dealing with a Pull Request, you'd have source and target branch), you can use Jenkins' Pipeline Syntax Snippet Generator, pick the Sample Step named 'checkout: Check out from version control', and under 'Additional Behaviours', click 'Add' and add the behaviour named 'Calculate changelog against a specific branch'. Then when you fill up the fields and click 'Generate Pipeline Script', you'll have the 'ChangelogToBranch' option configured for you (which will fill changeset accordingly). – Despain

Answer 3

Basically, you need two jobs. One to check whether files changed and one to do the actual build:

Job #1

This should be triggered on changes in your Git repository. It then tests whether the path you specify ("src" here) has changes and then uses Jenkins' CLI to trigger a second job.

export JENKINS_CLI="java -jar /var/run/jenkins/war/WEB-INF/jenkins-cli.jar"
export JENKINS_URL=http://localhost:8080/
export GIT_REVISION=`git rev-parse HEAD`
export STATUSFILE=$WORKSPACE/status_$BUILD_ID.txt

# Figure out, whether "src" has changed in the last commit
git diff-tree --name-only HEAD | grep src

# Exit with success if it didn't
$? || exit 0

# Trigger second job
$JENKINS_CLI build job2 -p GIT_REVISION=$GIT_REVISION -s

Job #2

Configure this job to take a parameter GIT_REVISION like so, to make sure you're building exactly the revision the first job chose to build.

Charentemaritime

What if two or more commits happened since the last build? I think you might miss changes in src since you're only examining the HEAD commit. – Celsacelsius

@AdamMonsen Right. But you can easily adapt the above script to whatever situation/condition you want to test against.. for example not diffing against HEAD but against what was HEAD last time the script ran. – Charentemaritime

There's something missing at $? || exit 0... test $? -eq 0 || exit 0 maybe? – Dicker

jenkins-cli.jar is disabled for security reason in my company, how can I specify the GIT-REVISION on the Repository URL in the Source Code Management section of jenkins? For SVN, we can suffix the url with something like @${repo_rev}. – Lapides

Looks like something I can put into the .when{} block of a pipeline to make it conditional, no? – Demythologize

Answer 4

While this doesn't affect single jobs, you can use this script to ignore certain steps if the latest commit did not contain any changes:

/*
 * Check a folder if changed in the latest commit.
 * Returns true if changed, or false if no changes.
 */
def checkFolderForDiffs(path) {
    try {
        // git diff will return 1 for changes (failure) which is caught in catch, or
        // 0 meaning no changes 
        sh "git diff --quiet --exit-code HEAD~1..HEAD ${path}"
        return false
    } catch (err) {
        return true
    }
}

if ( checkFolderForDiffs('api/') ) {
    //API folder changed, run steps here
}

Cohere

@Karl feel free to fix the code if it works for you. That was the one issue I had when applying this code (On build failures, it wouldn't retry this commit, if the absolute latest commit didn't also change the api/ folder.) If you can fix this, I would love a suggested change! – Cohere

Answer 5

For Bitbucket Repository users (and other people using Source-Control Management hosts which webhook payload doesn't seem to indicate file changes).

It seems that Git plugin 'included regions' fail whatever I do, and always trigger the job. My setup is Jenkins 2.268, run in a Docker container, and it was purgatory to find a correct way to achieve building jobs depending on file changes, but here's one below.

Required Jenkins plugins:

• Groovy
• Bitbucket (or, if you're on another SCM host: a plugin which can trigger builds on this host's webhooks)

Create a new Freestyle job called 'Switch':

1. Source Code Management: indicate your SCM information (make sure the 'Branches to build' are the right ones.
2. Build triggers > Build when a change is pushed to Bitbucket: checked
3. Build step > Execute system Groovy script (not just Execute Groovy script!), leaving Use Groovy sandbox unchecked.

The script:

import jenkins.*;
import jenkins.model.*;

// CONFIGURATION
// Links between changed file patterns and job names to build
def jobsByPattern = [
  "/my-project/": "my-project-job",
  "my-super-project/":"super-job",
]

// Listing changes files since last build
def changeLogSets = build.changeSets
def changedFiles = []
for (int i = 0; i < changeLogSets.size(); i++) {
   def entries = changeLogSets[i].items
   for (int j = 0; j < entries.length; j++) {
    def entry = entries[j]
    def files = new ArrayList(entry.affectedFiles)
    for (int k = 0; k < files.size(); k++) {
       def file = files[k]
       changedFiles.add(file.path)
    }
  }
}

// Listing ad hoc jobs to build
jobsToBuild = [:] // declare an empty map
for(entry in jobsByPattern ) {
  def pattern = entry.key
  println "Check pattern: $pattern"
  for (int i = 0; i < changedFiles.size(); i++) {
    def file = changedFiles[i]
    println "Check file: $file"
    if( file.contains( pattern ) ) {
      def jobName = entry.value
      jobsToBuild[ jobName ] = true
      break
    }
  }
}

// Building appropriate jobs
jobsToBuild.each{
  def jobName = it.key
  println "$jobName must be built!"
  def job = Jenkins.instance.getJob(jobName)
  def cause = new hudson.model.Cause.UpstreamCause(build)
  def causeAction = new hudson.model.CauseAction(cause)
  Jenkins.instance.queue.schedule(job, 0, causeAction)
}

I believe this method can handle multiple commits since the last build, so it seems to answer the need. Any suggestion of enhancement welcomed.

Unscramble

Answer 6

I wrote this script to skip or execute tests if there are changes:

#!/bin/bash

set -e -o pipefail -u

paths=()
while [ "$1" != "--" ]; do
    paths+=( "$1" ); shift
done
shift

if git diff --quiet --exit-code "${BASE_BRANCH:-origin/master}"..HEAD ${paths[@]}; then
    echo "No changes in ${paths[@]}, skipping $@..." 1>&2
    exit 0
fi
echo "Changes found in ${paths[@]}, running $@..." 1>&2

exec "$@"

So you can do something like:

./scripts/git-run-if-changed.sh cmd vendor go.mod go.sum fixtures/ tools/ -- go test

Vastah

Answer 7

If the logic for choosing the files is not trivial, I would trigger script execution on each change and then write a script to check if indeed a build is required, then triggering a build if it is.

Wessel

Answer 8

You can use Generic Webhook Trigger Plugin for this.

With a variable like changed_files and expression $.commits[*].['modified','added','removed'][*].

You can have a filter text like $changed_files and filter regexp like "folder/subfolder/[^"]+?" if folder/subfolder is the folder that should trigger builds.

Balsaminaceous

I'm trying to do this but I'm a little bit lost. how to send the path of the changed file to jenkins ? could you explain a little bit more please? Where to put the variable changed_files? – Oncoming

You need to configure a webhook in the Git service you are using. If it is GitHub there is an example here: github.com/jenkinsci/generic-webhook-trigger-plugin/blob/master/… – Balsaminaceous

In fact as I'm using Bitbucket, I realized that Changed_files entry is not available in the payload of push event in bibucket (ref: confluence.atlassian.com/bitbucket/… ) so I'm not sure how can I do this. I'll rely on the commit message I think. thank you – Oncoming

@Oncoming : exact same problem here. I found a way, which I posted here. Could it help? – Unscramble

Answer 9

I answered this question in another post:

How to get list of changed files since last build in Jenkins/Hudson

#!/bin/bash

set -e

job_name="whatever"
JOB_URL="http://myserver:8080/job/${job_name}/"
FILTER_PATH="path/to/folder/to/monitor"

python_func="import json, sys
obj = json.loads(sys.stdin.read())
ch_list = obj['changeSet']['items']
_list = [ j['affectedPaths'] for j in ch_list ]
for outer in _list:
  for inner in outer:
    print inner
"

_affected_files=`curl --silent ${JOB_URL}${BUILD_NUMBER}'/api/json' | python -c "$python_func"`

if [ -z "`echo \"$_affected_files\" | grep \"${FILTER_PATH}\"`" ]; then
  echo "[INFO] no changes detected in ${FILTER_PATH}"
  exit 0
else
  echo "[INFO] changed files detected: "
  for a_file in `echo "$_affected_files" | grep "${FILTER_PATH}"`; do
    echo "    $a_file"
  done;
fi;

You can add the check directly to the top of the job's exec shell, and it will exit 0 if no changes are detected... Hence, you can always poll the top level for check-in's to trigger a build.

Wollongong