Block a job from running if given node(s) with a given label(s) is/are running another job(s)

Question

In Jenkins, we can block a job A if job B is running using Build blocker plugin.

Similarly or in some fashion, I would like a job, for ex: another_dumb_job to NOT run / (wait and let it sit in queue) if there are any in-progress jobs running on any user selected slave(s) until those slaves are free again.

For ex: I don't want to run a Job (which will delete bunch of slaves either offline/online -- using a downstream job or via calling some groovy/scriptler script) until any of those slave(s) have active/in-progress job(s) running on them?

The end goal is to delete Jenkins node slaves gracefully i.e. the node/slave is marked OFFLINE first, then any existing jobs (running on a slave are complete) and then the slaves get deleted.

Subaltern

Perhaps you can use the wiki.jenkins-ci.org/display/JENKINS/… to create a category for those jobs, and allow only one at a time. – Puritanical

@Tizkiko thanks for sharing. I think it's to just set max # of a given job to run OR max# of any job to run on a given node/slave. In my case, I wanted the job to block/wait until all node slaves starting with a given name are done from running any already in-progress job on them. This was required in my case as I was creating those slaves on a machine(s) and was auto provisioning/tearing them out on the fly -- but didn't want the tearing part to happen until the node/slaves were sitting idle. – Subaltern

Answer 1

For deleting all offline nodes, tweak the script below and run doDelete() only on slaves where isOffline() is true or isOnline() is false. If you want to delete all nodes (be careful) then don't use the the following if statement:

if ( aSlave.name.indexOf(slaveStartsWith) == 0) {

I'm also ignoring a slave (if you want to ALWAYS ignore a slave from getting deleted). It can be enhanced to use a list of slaves to ignore.

Anyways, the following script will gracefully delete any Jenkins node slaves which starts with a given name (so that you have more control) and it'll mark offline (asap) but delete it only after any running job(s) on that given slave(s) is/are complete. Thought I should share here.

Using Jenkins Scriptler Plugin, one can import/upload/run this script: https://github.com/gigaaks/jenkins-scripts/blob/7eaf41348e886db108bad9a72f876c3827085418/scriptler/disableSlaveNodeStartsWith.groovy

/*** BEGIN META {
  "name" : "Disable Jenkins Hudson slaves nodes gracefully for all slaves starting with a given value",
  "comment" : "Disables Jenkins Hudson slave nodes gracefully - waits until running jobs are complete.",
  "parameters" : [ 'slaveStartsWith'],
  "core": "1.350",
  "authors" : [
    { name : "GigaAKS" }, { name : "Arun Sangal" }
  ]
} END META**/

// This scriptler script will mark Jenkins slave nodes offline for all slaves which starts with a given value.
// It will wait for any slave nodes which are running any job(s) and then delete them.
// It requires only one parameter named: slaveStartsWith and value can be passed as: "swarm-".

import java.util.*
import jenkins.model.*
import hudson.model.*
import hudson.slaves.*

def atleastOneSlaveRunnning = true;
def time = new Date().format("HH:mm MM/dd/yy z",TimeZone.getTimeZone("EST"))

while (atleastOneSlaveRunnning) {

 //First thing - set the flag to false.
 atleastOneSlaveRunnning = false; 
 time = new Date().format("HH:mm MM/dd/yy z",TimeZone.getTimeZone("EST"))

 for (aSlave in hudson.model.Hudson.instance.slaves) {

   println "-- Time: " + time;
   println ""
   //Dont do anything if the slave name is "ansible01"
   if ( aSlave.name == "ansible01" ) {
        continue;
   }  
   if ( aSlave.name.indexOf(slaveStartsWith) == 0) {
       println "Active slave: " + aSlave.name; 

       println('\tcomputer.isOnline: ' + aSlave.getComputer().isOnline());
       println('\tcomputer.countBusy: ' + aSlave.getComputer().countBusy());
       println ""
       if ( aSlave.getComputer().isOnline()) {
            aSlave.getComputer().setTemporarilyOffline(true,null);
            println('\tcomputer.isOnline: ' + aSlave.getComputer().isOnline());    
            println ""
       }
       if ( aSlave.getComputer().countBusy() == 0 ) {
            time = new Date().format("HH:mm MM/dd/yy z",TimeZone.getTimeZone("EST"))
            println("-- Shutting down node: " + aSlave.name + " at " + time);
            aSlave.getComputer().doDoDelete(); 
       } else {
            atleastOneSlaveRunnning = true;  
       }
  }
 }
 //Sleep 60 seconds  
 if(atleastOneSlaveRunnning) { 
   println ""
   println "------------------ sleeping 60 seconds -----------------"
   sleep(60*1000); 
   println ""   
 } 
}

Now, I can create a free-style jenkins job, use Scriptler script in Build action and use the above script to gracefully delete slaves starting with a given name (job parameter getting passed to scriptler script).

If you are fast enough to get the following error message, that means, you ran or called the Scriptler script (as shown above) in a job and restricted that job to run on a non-master aka node/slave machine. Scriptler Scripts are SYSTEM Groovy scripts i.e. they must run on Jenkins master's JVM to access all Jenkins resources/tweak them. To fix the following issue, you can create a job (restrict it to run on master server i.e. Jenkins master JVM) which will just accept one parameter for the scriptler script and call this job from the first job (as Trigger a project/job and block until the job is complete):

21:42:43 Execution of script [disableSlaveNodesWithPattern.groovy] failed - java.lang.NullPointerException: Cannot get property 'slaves' on null objectorg.jenkinsci.plugins.scriptler.util.GroovyScript$ScriptlerExecutionException: java.lang.NullPointerException: Cannot get property 'slaves' on null object
21:42:43    at org.jenkinsci.plugins.scriptler.util.GroovyScript.call(GroovyScript.java:131)
21:42:43    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
21:42:43    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
21:42:43    at hudson.remoting.Request$2.run(Request.java:328)
21:42:43    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
21:42:43    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
21:42:43    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
21:42:43    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
21:42:43    at java.lang.Thread.run(Thread.java:745)
21:42:43 Caused by: java.lang.NullPointerException: Cannot get property 'slaves' on null object

i.e. If you have Scriptler script build step running in a job (which is not running on a MASTER Jenkins machine/JVM), then the above error will come and to solve it, create a job "disableSlaveNodesStartsWith" and restrict it to run on master (safer side) and calling Scriptler script and pass parameter to the job/script.

Now, from the other job, call this job:

Subaltern