Jenkins 2.89.4 rolling
I saw almost all stackoverflow posts which show how we can successfully run parallel steps/stages (using list/maps etc) --OR hardcoding them directly --OR even create dynamic stages for Jenkinsfile (as seen in this post: Scripted jenkinsfile parallel stage)
My requirements are:
A pipeline which builds N. no of projects under "BUILD" steps i.e. parallel builds on each of those projects. i.e. it runs Gradle on all N projects. Here I have a Jenkinsfile which was created by a declarative JOB DSL Groovy. Here my Gradle projects are not set as multi-projects so I can't call the top level gradle and say, Gradle please do your parallel magic (within Gradle).
I want to run build of these N projects in their own separate parallel dynamically created stages (GUI Columns) as seen in jenkins job's dashboard.
I want to see the output of (Gradle build/console) each project's build separately i.e. I don't want to mix the console output of each projects build which are running in parallel in just ONE COLUMN (i.e. column named
BUILD).In this URL https://jenkins.io/blog/2017/09/25/declarative-1/ I see, how you can run parallel stages/steps but in doing so, either it's mixing those parallel step's output in just one column (I mean under BUILD column) --OR if you want it under separate stages/columns (i.e. the post says Test on Linux or Windows separately then you are still hard-coding all the stages/steps in Jenkinsfile early on (rather than using just a list or array hash which I'd prefer to update for adding more or less stages / parallel steps as in my case, they all follow the same standard). What I want is to just update in one place How many steps and what are all of the stages in just one place (list/array).
- I'm not using Jenkins Blue Ocean for now.
Usually if you have parallel steps within a stage, their console std output for all steps gets mixed into one console output / stage/column when you click to see the console output for that given parallel step/stage; When you hover over the BUILD column (assuming there were parallel steps in BUILD stage) in job's dashboard (std output for all those steps is mixed up and very hard to see individual project step's console output just for a given step/stage).
If we want to create separate stages (dynamically) then Jenkins should be able to show console output of a given step/dynamic stage within the parallel section (i.e. each column should show their own project's build console output).
Using the above URL, I'm able to do the following after trying this script:
// main script block
// could use eg. params.parallel build parameter to choose parallel/serial
def runParallel = true
def buildStages
node('master') {
stage('Initializing Parallel Dynamic Stages') {
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
println("Initialised pipeline.")
}
for (builds in buildStages) {
if (runParallel) {
parallel(builds)
} else {
// run serially (nb. Map is unordered! )
for (build in builds.values()) {
build.call()
}
}
}
stage('Done') {
println('The whole SHENZI is complete.')
}
}
// Create List of build stages to suit
def prepareBuildStages() {
def buildList = []
for (i=1; i<4; i++) {
def buildStages = [:]
for (name in [ 'Alpha', 'Tango', 'Chyarli' ] ) {
def n = "${name} ${i}"
buildStages.put(n, prepareOneBuildStage(n))
}
buildList.add(buildStages)
}
return buildList
}
def prepareOneBuildStage(String name) {
def proj_name = name.split(' ')[0]
def proj_parallel_sub_step = name.split(' ')[1]
//Return the whole chunkoni
return {
stage("Build\nProject-${proj_name}\nStep ${proj_parallel_sub_step}") {
println("Building ${proj_name} - ${proj_parallel_sub_step}")
sh(script:'sleep 15', returnStatus:true)
}
}
}
When I'm putting the above Groovy Script (which is creating DYNAMIC Stages) inside Pipeline Script or Pipeline Script from SCM (i.e. the same code available in a .groovy file) -- it runs successfully and creates dynamic stages under BUILD step for each of the 3 projects and runs 3 steps (Nth) for all 3 projects in parallel and then starts the next Nth step for all 3 projects and so on.
If you see below, we also got individual columns in Jenkins job dashboard for them.
Now, When I put the above script in Jenkinsfile (Pipeline DSL) where I have pipeline { .... } section, it's not working and giving me the following error.
Using my JOB DSL, I created a new Jenkins Pipeline job where Pipeline Script from SCM calls a groovy file (which now contains):
//----------------------------------------------------
// Both - Parallel Run and GUI View in JF Jenkins job.
//----------------------------------------------------
def runParallel = true
def buildStages
def wkspace = /var/lib/jenkins/workspaces/ignore_this_variale_or_its_value_for_now
// Create List of build stages to suit
def prepareBuildStages() {
def buildList = []
for (i=1; i<3; i++) {
def buildStages = [:]
for (name in [ 'Alpha', 'Tango', 'Chyarli' ] ) {
def n = "${name} ${i}"
buildStages.put(n, prepareOneBuildStage(n))
}
buildList.add(buildStages)
}
return buildList
}
//---
def prepareOneBuildStage(String name) {
def proj_name = name.split(' ')[0]
def proj_parallel_sub_step = name.split(' ')[1]
// return the whole chunkoni (i.e. for a given stage) - will be named dynamically.
return {
stage("Build\nProject-${proj_name}\nStep ${proj_parallel_sub_step}") {
println("Building ${proj_name} - ${proj_parallel_sub_step}")
sh(script:'sleep 15', returnStatus:true)
}
}
}
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
//---------------------
String jenkinsBaselines
// SEE NOW --- we have this section called 'pipeline'
pipeline {
agent {
node {
label 'rhat6'
customWorkspace wkspace
}
}
options {
ansiColor('xterm')
timeout(time: 8, unit: 'HOURS')
skipDefaultCheckout()
timestamps()
}
environment {
someEnvVar = 'aValue'
}
//------------- Stages
stages {
stage('Initializing Parallel Dynamic Stages') {
// Set up List<Map<String,Closure>> describing the builds
println("Initialised pipeline.")
}
for (builds in buildStages) {
if (runParallel) {
parallel(builds)
} else {
// run serially (nb. Map is unordered! )
for (build in builds.values()) {
build.call()
}
}
}
stage('Done') {
println('The whole SHENZI is complete.')
}
}
//---------------------
}
Running the Jenkinsfile Jenkins job now gives me this error:
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 1s
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 69: Not a valid stage section definition: "buildStages = prepareBuildStages()". Some extra configuration is required. @ line 69, column 5.
stage('Initializing Parallel Dynamic Stages') {
^
WorkflowScript: 69: Unknown stage section "println". Starting with version 0.5, steps in a stage must be in a steps block. @ line 69, column 5.
stage('Initializing Parallel Dynamic Stages') {
^
WorkflowScript: 75: Expected a stage @ line 75, column 5.
for (builds in buildStages) {
^
WorkflowScript: 86: Unknown stage section "println". Starting with version 0.5, steps in a stage must be in a steps block. @ line 86, column 5.
stage('Done') {
^
WorkflowScript: 69: No "steps" or "parallel" to execute within stage "Initializing Parallel Dynamic Stages" @ line 69, column 5.
stage('Initializing Parallel Dynamic Stages') {
^
WorkflowScript: 86: No "steps" or "parallel" to execute within stage "Done" @ line 86, column 5.
stage('Done') {
^
6 errors
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE
How can I get this working in Jenkinsfile pipeline section and still able to get individual columns per dynamically created stage for a given project N and Step M?
Tried the following way, still errors the say way.
//------------- Stages
stages {
stage('Initializing Parallel Dynamic Stages') {
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
println("Initialised pipeline.")
// tried this way too. within a stage
buildStages.each { bld -->
parallel(bld)
}
}
stage('Done') {
println('The whole SHENZI is complete.')
}
}
//---------------------