Scripted jenkinsfile parallel stage

Question

I am attempting to write a scripted Jenkinsfile using the groovy DSL which will have parallel steps within a set of stages.

Here is my jenkinsfile:

node {   
stage('Build') {
    sh 'echo "Build stage"'
}

stage('API Integration Tests') {
    parallel Database1APIIntegrationTest: {
        try {
            sh 'echo "Build Database1APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }               

    }, Database2APIIntegrationTest: {
        try {
            sh 'echo "Build Database2APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }

    }, Database3APIIntegrationTest: {
        try {
            sh 'echo "Build Database3APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }
    }
}

stage('System Tests') {
    parallel Database1APIIntegrationTest: {
        try {
            sh 'echo "Build Database1APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }               

    }, Database2APIIntegrationTest: {
        try {
            sh 'echo "Build Database2APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }

    }, Database3APIIntegrationTest: {
        try {
            sh 'echo "Build Database3APIIntegrationTest parallel stage"'
        }
        finally {
            sh 'echo "Finished this stage"'
        }
    }
}
}

I want to have 3 stages: Build; Integration Tests and System Tests. Within the two test stages, I want to have 3 sets of the tests executed in parallel, each one against a different database.

I have 3 available executors. One on the master, and 2 agents and I want each parallel step to run on any available executor.

What I've noticed is that after running my pipeline, I only see the 3 stages, each marked out as green. I don't want to have to view the logs for that stage to determine whether any of the parallel steps within that stage were successful/unstable/failed.

I want to be seeing the 3 steps within my test stages - marked as either green, yellow or red (Success, unstable or failed).

I've considered expanding the tests out into their own stages, but have realised that parallel stages are not supported (Does anyone know whether this will ever be supported?), so I cannot do this as the pipeline would take far too long to complete.

Any insight would be much appreciated, thanks

Walleye

Answer 1

In Jenkins scripted pipeline, parallel(...) takes a Map describing each stage to be built. Therefore you can programatically construct your build stages up-front, a pattern which allows flexible serial/parallel switching.
I've used code similar to this where the prepareBuildStages returns a List of Maps, each List element is executed in sequence whilst the Map describes the parallel stages at that point.

// main script block
// could use eg. params.parallel build parameter to choose parallel/serial 
def runParallel = true
def buildStages

node('master') {
  stage('Initialise') {
    // Set up List<Map<String,Closure>> describing the builds
    buildStages = prepareBuildStages()
    println("Initialised pipeline.")
  }

  for (builds in buildStages) {
    if (runParallel) {
      parallel(builds)
    } else {
      // run serially (nb. Map is unordered! )
      for (build in builds.values()) {
        build.call()
      }
    }
  }

  stage('Finish') {
      println('Build complete.')
  }
}

// Create List of build stages to suit
def prepareBuildStages() {
  def buildStagesList = []

  for (i=1; i<5; i++) {
    def buildParallelMap = [:]
    for (name in [ 'one', 'two', 'three' ] ) {
      def n = "${name} ${i}"
      buildParallelMap.put(n, prepareOneBuildStage(n))
    }
    buildStagesList.add(buildParallelMap)
  }
  return buildStagesList
}

def prepareOneBuildStage(String name) {
  return {
    stage("Build stage:${name}") {
      println("Building ${name}")
      sh(script:'sleep 5', returnStatus:true)
    }
  }
}

The resulting pipeline appears as:

There are certain restrictions on what can be nested within a parallel block, refer to the pipeline documentation for exact details. Unfortunately much of the reference seems biased towards declarative pipeline, despite it being rather less flexible than scripted (IMHO). The pipeline examples page was the most helpful.

Winton

Is it possible to have multiple steps in a parallel execution? What I mean: is is possible to have {<one1.1, one1.2> || <two1> || <three1.1, three1.2, three1.3>} as a parallel step, where the 1.n are non-parallel stages? – Slobbery

@D.Kovács I think you can only have one stage{} since it's the value of the map. But you can certainly put multiple steps within that stage{}, where a step is one of jenkins.io/doc/pipeline/steps as described in jenkins.io/doc/book/pipeline/syntax/#scripted-steps Possibly the Jenkins/BlueOcean UI would display them as just a single dot. – Winton

This code works if I use it inside 'Pipeline Script", but not when I use "Pipeline Script from SCM" (where I have the same code in a .groovy file - coming from GIT/BitBucket) when I wrap it within pipeline { ... } section. How can I get that solved and also get the "Multi-columns" created for each Dynamically created "Stages" when I click on the Jenkinsfile job's dashboard (GUI). PS: Im not using Blue Ocean for now – Halfsole

@ArunSangal when you wrap in pipeline{ ... } you're not using scripted pipeline, you're using pipeline DSL. Someone thought it would be a good idea to create an easier way to write pipelines. But there are so many limitations and gotcha's you might as well ignore it and go with scripted syntax IMNSHO. – Winton

@EdRandall So I tried it over the weekend and I think I got it (hello world way for now). Your post helped too. I'm tracking it here: #58886345 (when in pipeline { ... } section) – Halfsole

@EdRandall Great post! A pity a doc on parallel scripting is missing in the Jenkins pipelines. cookbook There is some confusion here with your return parameters. I would rename the output of prepareBuildStages() to buildStages or rename the variable within the Initialise stage ;-) – Sold

Note that you can't directly use the loop variable inside the closure, as it is captured by reference (name in this example): #56348841 – Piloting

See also jenkins.io/doc/pipeline/examples/#parallel-from-list for more examples. – Piloting

+1 for bias towards declarative pipeline, scripting capabilities is somethign what makes jenkins unique among many others and they don't value it. – Mercorr

Answer 2

Here's a simple example without loops or functions based on @Ed Randall's post:

node('docker') {
    stage('unit test') {
        parallel([
            hello: {
                echo "hello"
            },
            world: {
                echo "world"
            }
        ])
    }

    stage('build') {
        def stages = [:]

        stages["mac"] = {
            echo "build for mac"
        }
        stages["linux"] = {
            echo "build for linux"
        }

        parallel(stages)
    }
}

...which yields this:

Note that the values of the Map don't need to be stages. You can give the steps directly.

Logo

This works nicely but is there any way to use failFast or something similar in scripted pipelines? – Reconnoiter

@Fjaoos, I haven't tested it but a quick google gives some hints: #58026689 also, the jenkins pipeline snippet generator has some advice: parallel firstBranch: { // do something }, secondBranch: { // do something else }, failFast: true|false – Logo

Answer 3

Here is an example from their docs:

Parallel execution

The example in the section above runs tests across two different platforms in a linear series. In practice, if the make check execution takes 30 minutes to complete, the "Test" stage would now take 60 minutes to complete!

Fortunately, Pipeline has built-in functionality for executing portions of Scripted Pipeline in parallel, implemented in the aptly named parallel step.

Refactoring the example above to use the parallel step:

// Jenkinsfile (Scripted Pipeline)


stage('Build') {
    /* .. snip .. */
}

stage('Test') {
    parallel linux: {
        node('linux') {
            checkout scm
            try {
                unstash 'app'
                sh 'make check'
            }
            finally {
                junit '**/target/*.xml'
            }
        }
    },
    windows: {
        node('windows') {
            /* .. snip .. */
        }
    }
}

Biceps

parallel in stage level is something a bit different then parallel in script level. Sadly on my Jenkins machine parallel in stage level ends with stackoverflow :(. – Oudh

Answer 4

To simplify the answer of @Ed Randall here. Remember this is Jenkinsfile scripted (not declarative)

stage("Some Stage") {
    // Stuff ...
}


stage("Parallel Work Stage") {

    // Prealocate dict/map of branchstages
    def branchedStages = [:]

    // Loop through all parallel branched stage names
    for (STAGE_NAME in ["Branch_1", "Branch_2", "Branch_3"]) {

        // Define and add to stages dict/map of parallel branch stages
        branchedStages["${STAGE_NAME}"] = {
            stage("Parallel Branch Stage: ${STAGE_NAME}") {
                // Parallel stage work here
                sh "sleep 10"
            }
        }

    }

    // Execute the stages in parallel
    parallel branchedStages
}


stage("Some Other Stage") {
    // Other stuff ...
}

Please pay attention to the curly braces. This will result in the following result (with the BlueOcean Jenkins Plugin):

Gelatinate

There might be an issue? In my test, the code stage("Parallel Branch Stage: ${STAGE_NAME}") always use the last one in the list, because STAGE_NAME is a variable outside of the closure. – Pokeweed

@WeiHuang Yes, you can't directly use the loop variable, as the closure will capture the variable, not the content. See #56348841 – Piloting

Can you please help me to figure out the solution to similar problem ? #76407500 – Teenyweeny

Answer 5

I was also trying similar sort of steps to execute parallel stages and display all of them in a stage view. You should write a stage inside a parallel step as shown in the following code block.

// Jenkinsfile (Scripted Pipeline)

stage('Build') {
    /* .. Your code/scripts .. */
}

stage('Test') {
    parallel 'linux': {
        stage('Linux') {
            /* .. Your code/scripts .. */
        }
    }, 'windows': {
        stage('Windows') {
            /* .. Your code/scripts .. */
        }
    }
}

Polyclinic

Answer 6

The above example with a FOR is wrong, as varible STAGE_NAME will be overwritten everytime, I had the same problem as Wei Huang.

Found the solution here:

https://www.convalesco.org/notes/2020/05/26/parallel-stages-in-jenkins-scripted-pipelines.html

def branchedStages = [:]
def STAGE_NAMES =  ["Branch_1", "Branch_2", "Branch_3"]
STAGE_NAMES.each { STAGE_NAME ->
 // Define and add to stages dict/map of parallel branch stages
    branchedStages["${STAGE_NAME}"] = {
        stage("Parallel Branch Stage: ${STAGE_NAME}") {
        // Parallel stage work here
            sh "sleep 10"
        }
    }
  }
parallel branchedStages

Coccidioidomycosis

Answer 7

I have used as below where the three stages are parallel.

def testCases() {
  stage('Test Cases') {
    def stages = [:]    // declaring empty list
      stages['Unit Testing'] = {
      sh "echo Unit Testing completed"
      }
      stages['Integration Testing'] = {
        sh "echo Integration Testing completed"
      }
      stages['Function Testing'] = {
        sh "echo Function Testing completed"
      }
    parallel(stages) // declaring parallel stages
  }
}   

Iatric

Answer 8

I have used stage{} in parallel blocks several times. Then each stage shows up in the Stage view. The parent stage that contains parallel doesn't include the timing for all the parallel stages, but each parallel stage shows up in stage view.

In blue ocean, the parallel stages appear separately instead of the stages showing. If there is a parent stage, it shows as the parent of the parallel stages.

If you don't have the same experience, maybe a plugin upgrade is due.

Smoky

Can you give an example of where you have used a stage within parallel blocks? I am struggling to find any documentation on this – Walleye