Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?

Question

I have a list of long running Gradle tasks on different sub projects in my project. I would like to run these in parallel using Jenkins declarative pipeline.

I was hoping something like this might work:

projects = [":a", ":b", ":c"]

pipeline {
    stage("Deploy"){
        parallel {
             for(project in projects){
               stage(project ) {
                   when {
                       expression {
                            someConditionalFunction(project)
                       }
                   }
                   steps {
                       sh "./gradlew ${project}:someLongrunningGradleTask"
                  }
                }   
             }
        }
    }
}

Needless to say that gives a compile error since it was expecting stage instead of for. Any ideas on how to overcome this? Thanks

Natishanative

Answer 1

I was trying to reduce duplicated code in my existing Jenkinsfile using declarative pipeline syntax. Finally I was able to wrap my head around the difference between scripted and declarative syntax.

It is possible to use scripted pipeline syntax in a declarative pipeline by wrapping it with a script {} block.

Check out my example below: you will see that all three parallel stages finish at the same time after waking up from the sleep command.

def jobs = ["JobA", "JobB", "JobC"]

def parallelStagesMap = jobs.collectEntries {
    ["${it}" : generateStage(it)]
}

def generateStage(job) {
    return {
        stage("stage: ${job}") {
                echo "This is ${job}."
                sh script: "sleep 15"
        }
    }
}

pipeline {
    agent any

    stages {
        stage('non-parallel stage') {
            steps {
                echo 'This stage will be executed first.'
            }
        }

        stage('parallel stage') {
            steps {
                script {
                    parallel parallelStagesMap
                }
            }
        }
    }
}

Revealment

@mkobit I see with Declarative Pipeline 1.2 I can add pipeline code to Shared library, but I cannot add method to under def call How can I add all above code in Shared Library including def part – Cassette

I added pipeline part into Shared Library and I'm calling it like @Library('pipeline_lib') _ parallelDeclare() . But I get error groovy.lang.MissingPropertyException: No such property: parallelStagesMap for class: parallelDeclare – Cassette

I would add more to this answer. How to throttle parallel stages: issues.jenkins-ci.org/browse/… – Beak

though there are parallel jobs running,i see only 1 job that actually run and gives the output and other jobs that are created in parallel just waits. But on a 'ps -ef' i see multiple jobs running in parallel but internally waiting for 1 job (the last job in parallelStageMap) to complete. Should i use throttling to make all the parallel job run the actual task ? – Hydrology

for me this uses scripted pipeline stages; when you add agent, when, tools, or similar declarative constructs into generateStage it fails :( – Gregson

This article successfully uses declarative stages inside parallel with pipelines 1.2, still it remains unclear how to generate stages programmatically: jenkins.io/blog/2017/09/25/declarative-1 – Gregson

@Gregson did you find a solution to this issue ? – Marmoset

No, I still use the imperative parallel approach. – Gregson

😔 ]It runs each Job in different branch !? It translates the key of the Map entry ("${it}" ) as the branch name, and the value of the Map entry (generateStage(it)) as the stage to run for that branch. – Rihana

@Gregson any news about "generate stages programmatically" ?? – Rihana

@AbdennourTOUMI, no, already reported at Sep 5th that I still use the imperative approach :( – Gregson

Instead of making parallelStagesMap a map, is there way to make it a function that returns a map? I tried it, but when the build runs, the stage fails without any error logs. :( – Discretionary

Works like a charm, in the declarative pipeline! Thank you!! – Xeniaxeno

Answer 2

Parallel wants a map structure. You are doing this a little inside-out. Build your map and then just pass it to parallel, rather than trying to iterate inside parallel.

Option 2 on this page shows you a way to do something similar to what you are trying.

At this link you can find a complex way I did this similar to a matrix/multi-config job:

Dipper

Cheers for the response and links Rob. Is that possible for the declarative pipeline too? Im using the syntax here: jenkins.io/doc/book/pipeline/syntax/#parallel It may be the case that its only possible using scripted pipeline – Natishanative

You would probably need to create the map in a script{} block or outside of the pipeline{} block, but yes, you can still do the same thing in declarative. You just have to combine it with a little scripted. The two merge together really well once you understand the variable scoping. – Dipper

@RobHales I have similar question but couldn't exactly figure it out can you please give a quick look #52086602 – Cassette