Jenkins declarative pipeline: Sequential execution of an axis of a matrix-build

Question

I am trying to set up a matrix build and am running in several issues. The matrix looks like:

stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
            axis {
                name 'MODULES'
                values 'repo1/module1', 'repo1/module2', 'repo2/module1', 'repo2/module2'
            }
        }

        stages {                                                
            stage( 'Checkout' ) {
            }
            stage( 'Build' ) {
            }
            stage( 'Test' ) {
            }
       }

The issue I have:

• jenkins is executing every cell of the matrix in it's own workspace but my modules depend on each other. That's why I want do check them out and build them in the same workspace. BTW: my build system is made to work with all the variants (debug/release x vc9/vc19) in the same workspace.

• jenkins is executing all the cells in parallel. What I need, is a serialization of execution of the MODULES axis.

Any ideas how I can work around this?

E.g. is there a way to descripe something like a loop over several modules that generates a sequence of stages in a row, not parallel? Within that sequence I could realize the matrix over 2 axes only.

I am aware of extended workspace plugin but did not find any documentation of how to use this in declarative pipelines.

Many thanks in advance! Regards, Christoph

Mazda

I have the same problem: I have 2 axis and I want to run first all cells for 1st entry of the axis and after they are completed I want to run all cells for the 2nd entry in the axis. But I don't see that this is possible – Cloudscape

I have this problem and i do not find any good solution. – Althing

Answer 1

You might be able to do what you want by using locks.

In my case, I just wanted the matrix to build sequentially instead of in parallel, so I just wrapped everything in a stage and put a lock on it. Something like:

stage( 'BuildAll' ) {
    matrix {
        axes {
            ...
        }
        stages {                                                
            stage( 'Sequential Matrix' ) {
                options {
                    lock( 'synchronous-matrix' )
                }
                stages {
                    ...
                }
            }
        }
    }
}

Only one matrix configuration can build at a time since the others can't acquire the lock until the current config is done.

In your case though, you want the modules to build one after the other in order. The string for the lock resource can be dynamic, so you can use a name constructed from the other 2 axes to prevent multiple modules from building at the same time (please see note below, this example doesn't work):

stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
            axis {
                name 'MODULES'
                values 'repo1/module1', 'repo1/module2', 'repo2/module1', 'repo2/module2'
            }
        }

        stages {                                                
            stage( 'Force Sequential' ) {
                options {
                    lock( "synchronous-${PLATFORM}-${VARIANT}" )
                }
                stages {
                    stage( 'Checkout' ) {

                    }
                    stage( 'Build' ) {

                    }
                    stage( 'Test' ) {

                    }
                }
            }
        }
    }
}

Note that this would only guarantee modules of the same platform and variant don't build at the same time and doesn't say anything about ordering. I don't know if the order which Jenkins chooses to build axes is defined, but I found matrix-sorter-plugin which may help with ordering if it's an issue.

IMPORTANT NOTE: The 2nd example doesn't work because of a groovy.lang.MissingPropertyException which AI copilot tells me is due to options being evaluated before the definition of matrix axes. I leave it as inspiration to try and come up with a working solution.

Antananarivo

I like this approach a lot because I'm trying to have the matrix run parallel builds per PLATFORM only. However, I'm getting this error when I put a lock on PLATFORM is: "no such property PLATFORM" and when I remove the lock it just works in full parallel mode. Any ideas? – Lumberjack

Sorry, late response, but if it's still relevant, could you share the code? – Antananarivo

The second example (which looks great) does not work (groovy.lang.MissingPropertyException: No such property: PLATFORM for class: groovy.lang.Binding ). Probably options can't be "dynamic", which sucks... – Hanyang

Apologies, Hcorg and Mirza, you're both correct. According to AI copilot, my 2nd example doesn't work because options are evaluated before matrix variables are defined. I edited my answer to move lock into the individual stages themselves. Can you see if that works? – Antananarivo

Ah, but I realize by moving the lock into the stages, at the end of the stage, the lock is released and another module is able to grab it before going to the next stage, which makes it useless for the goal we're trying to achieve. I added a disclaimer to my answer. – Antananarivo

Answer 2

I used for loop instead of matrix and axis syntax. My workaround is not as pretty as using matrix syntax, but it does what it supposed to do.

String[] platformList = ['win32vc9', 'win32vc19']
String[] variantList = ['debug', 'release']

pipeline{
    agent any
    stages {
        stage('Deploy'){
            steps{
                script{
                    for(platform in platformList){
                        for(variant in variantList){
                            stage("Checkout"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Checkout" """
                            }
                            stage("Build"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Build" """
                            }
                            stage("Test"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Test" """
                            }
                        }
                    }
                }
            }
        }
    }
}

Andra

Answer 3

Declarative Pipeline matrix directive doesn't currently support sequential execution.
The best you can do in in the current matrix syntax is this:

 stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
        }
        stages {                
            stage('repo1/module1') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo1/module2') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo2/module1') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo2/module2') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
        }
    }
}

Others have asked for similar. I've filed JENKINS-62085 based on this question.

Dispatcher