Maven lifecycle within jenkins pipeline - how to best separate responsibilities?

Question

When working with jenkins 2 (declarative) pipelines and maven I always have a problem with how to organize things within the pipeline to make it resusable and flexible.

On the one side I would like to seperate the pipepline into logical stages like:

pipeline
 {
  stages
   {
    stage('Clean') {}
    stage('Build') {}
    stage('Test') {}
    stage('Sanity check') {}
    stage('Documentation') {}
    stage('Deploy - Test') {}
    stage('Selenium tests') {}
    stage('Deploy - Production') {}
    stage('Deliver') {}
   }
 }

On the other hand I have maven which runs with

mvn clean deploy site

Simply I could split up maven to

mvn clean
mvn deploy
mvn site

But the 'deploy' includes all lifecycle phases from

• validate
• compile
• test
• package
• verify
• install
• deploy

So I saw a lot of pipline examples which do things like

sh 'mvn clean compile'

and

sh 'mvn test'

which results in repeating the validate and compile step a second time and waste "time/resources" in this way. This could be resolved with doing a

sh 'mvn surefire:test'

instead of running the whole lifecycle again.

So my question is - which is the best way to get a good balance between the jenkins pipline stages and the maven lifecycle? For me I see two ways:

1. Split up the maven lifecycles to as much pipeline stages as possible - which will result in better jenkins user feedback (see which stage fails etc.)
2. Let maven do everything and use the jenkins pipeline only to work with the results of maven (i.e. analyzing unit test results etc.)

Or did I missunderstand something in the CI/CD practice?

Pieplant

Answer 1

Two month later I think I have a well balanced Jenkins pipeline script that is not complete, but works stable on windows and linux. It avoids pitfalls of other examples I have seen.

Jenkinsfile

pipeline
 {
  agent any

  tools
   {
    maven 'Maven3'
    jdk 'JDK8'
   }

  options
   {
    buildDiscarder(logRotator(numToKeepStr: '4'))
    skipStagesAfterUnstable()
    disableConcurrentBuilds()
   }


  triggers
   {
    // MINUTE HOUR DOM MONTH DOW
    pollSCM('H 6-18/4 * * 1-5')
   }


  stages
   {
    stage('Clean')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode clean'
           }
          else
           {
            bat 'mvn --batch-mode clean'
           }
         }
       }
     }

    stage('Build')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode compile'
           }
          else
           {
            bat 'mvn --batch-mode compile'
           }
         }
       }
     }

    stage('UnitTests')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode resources:testResources compiler:testCompile surefire:test'
           }
          else
           {
            bat 'mvn --batch-mode resources:testResources compiler:testCompile surefire:test'
           }
         }
       }
      post
       {
        always
         {
          junit testResults: 'target/surefire-reports/*.xml'
         }
       }
     }

    stage('Sanity check')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode checkstyle:checkstyle pmd:pmd pmd:cpd com.github.spotbugs:spotbugs-maven-plugin:spotbugs'
           }
          else
           {
            bat 'mvn --batch-mode checkstyle:checkstyle pmd:pmd pmd:cpd com.github.spotbugs:spotbugs-maven-plugin:spotbugs'
           }
         }
       }
     }

    stage('Packaging')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode jar:jar'
           }
          else
           {
            bat 'mvn --batch-mode jar:jar'
           }
         }
       }
     }

    stage('install local')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode jar:jar source:jar install:install'
           }
          else
           {
            bat 'mvn --batch-mode jar:jar source:jar install:install' // maven-jar-plugin falseCreation default is false, so no doubled jar construction here, but required for maven-install-plugin internal data
           }
         }
       }
     }

    stage('Documentation')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode site'
           }
          else
           {
            bat 'mvn --batch-mode site'
           }
         }
       }
      post
       {
        always
         {
          publishHTML(target: [reportName: 'Site', reportDir: 'target/site', reportFiles: 'index.html', keepAll: false])
         }
       }
     }

    stage('Deploy test')
     {
      steps
       {      
        script
         {
          if (isUnix()) 
           {
            // todo
           }
          else
           {
            bat returnStatus: true, script: 'sc stop Tomcat8'
            sleep(time:30, unit:"SECONDS")
            bat returnStatus: true, script: 'C:\\scripts\\clean.bat'
            bat returnStatus: true, script: 'robocopy "target" "C:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\webapps" Test.war'
            bat 'sc start Tomcat8'
            sleep(time:30, unit:"SECONDS")
           }
         }
       }
     }

    stage('Integration tests')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode failsafe:integration-test failsafe:verify'
           }
          else
           {
            bat 'mvn --batch-mode failsafe:integration-test failsafe:verify'
           }
         }
       }
     }

   }

 }

Hopefully this is interesting for other developers outside there.

I will update this here when I significantly improve it over time.

For those who also wish to see a maven pom along with a Jenkinsfile please have a look at my small example project at github: TemplateEngine

Pieplant

I wonder if this plays out well in multimodule scenarios? Especially the packaging stage seems a bit specific. – Phytophagous

@Phytophagous I am sorry but multiplatform and multimodule is still on my todo list - so it has not been tested yet for these scenarios. – Pieplant

Don't misunderstand me, I really appreciated your question and your answer.But I am faced with deeply nested multimodule projects, and I fathom, calling Maven in each stage all over again would lead to some inefficiencies. On the other hand, doing a complete lifecycle in one stage seems to be a bit off the point... – Phytophagous

@Phytophagous Please let me know what is inefficient from your point of view. From my point of view my solution is more efficient than most examples I have seen on the net, because these examples repeat the compile/test etc. maven phases over again - this has been eliminated by doing the correct calls. But if there are more inefficiencies, I am interested in. – Pieplant

Answer 2

I think there is no right answer, but the following example worked for us.

stage('Build and Unit Test') {
    mvn clean deploy -> with unit tests, without integration tests, deploy local

    deploy local:
    You can define in a maven profile the distributionManagement like:
    <distributionManagement>
        <repository>
            <id>localFile</id>
            <url>file:target/repository/</url>
        </repository>
        <snapshotRepository>
            <id>localFile</id>
            <url>file:target/repository/</url>
        </snapshotRepository>
    </distributionManagement>
}   

stage('Pre Integration Tests') {
    The binaries are now in target/repository.
    From there you can use the binaries as you like.
    Copy them to a server, deploy them on an application server, etc.
}

stage('Integration Tests') {
    maven failsafe:integration-test failsafe:verify
    Already all tests are compiled, just execute them and verify the result.
}

stage('Deploy to Binary Repository (Nexus, Artifactory, etc)') {
    Now if everything is ok, finally upload the Binaries.
    For that we use wagon-maven-plugin
    So from target/repository the files are uploaded to the Binary Repository.
}

So to wrap this up:

• Fail fast. If a unit test has errors -> fail the build.
• Only build once. Use the same binaries for test, deployment/integration test, upload to repository, etc.
• With that the stages are logical units, which gives you enough feedback where to look for errors.

Guzman