Jenkins pipeline sh does not seem to respect pipe in shell command

Question

I am using a Jenkinsfile in a pipeline on version 2.32.2.

For various reasons I want to extract the version string from the pom. I was hoping I wouldn't have to add the maven help plugin and use evaluate.

I quickly came up with a little sed expression to get it out of the pom which uses pipes and works on the commandline in the jenkins workspace on the executor.

$ sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g' 1.0.0-SNAPSHOT

It could probably be optimized, but I want to understand why the pipeline seems to be failing on piped sh commands. I've played with various string formats and am currently using a dollar slashy string.

The pipeline step looks like the following to allow for easy output of the command string:

script {
    def ver_script = $/sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'/$
    echo "${ver_script}"
    POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
    echo "${POM_VERSION}"
}

When run in the jenkins pipeline I get the following console output where it seems to be separating the piped commands into separate commands:

[Pipeline] script
[Pipeline] {
[Pipeline] echo
sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
[Pipeline] sh
[FRA-198-versioned-artifacts-44SD6DBQOGOI54UEF7NYE4ECARE7RMF7VQYXDPBVFOHS5CMSTFLA] Running shell script
+ sed -n /<version>/,/<version/p pom.xml
+ head -1
+ sed s/[[:blank:]]*<\/*version>//g
sed: couldn't write 89 items to stdout: Broken pipe
[Pipeline] }
[Pipeline] // script

Any guidance out there on how to properly use piped commands in a jenkinsfile ?

Negris

Answer 1

I know this kind of late answer, but whoever you who needs the solution without eval you can use /bin/bash -c "script" to make pipe works

script {
    POM_VERSION = sh(script: "/bin/bash -c 'sed -n \'/<version>/,/<version/p\' pom.xml | head -1 | sed \'s/[[:blank:]]*<\/*version>//g\'\''", returnStdout: true)
    echo "${POM_VERSION}"
}

The only problem with this method is hellish escape yet this way the subshell of pipe will be handled by our boy /bin/bash -c

Okeechobee

Answer 2

I finally put some thought into it and realized that pipe subshells are probably causing the issue. I know some of the evils of eval but I ended up wrappping this in an eval:

script {
    def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
    echo "${ver_script}"
    POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
    echo "${POM_VERSION}"
}   

Negris

Answer 3

If your environment allows it, I've found a simple solution to this problem to be to place your script containing pipes into a file, and then run that with sh, like so:

script.sh:

#!/bin/sh
kubectl exec --container bla -i $(kubectl get pods | awk '/foo-/{ print $1 }') -- php /code/dostuff

Jenkinsfile:

stage('Run script with pipes') {
  steps {
    sh "./script.sh"
  }
}

Dearman

Answer 4

The pipeline-utility-steps plugin nowadays includes a readMavenPom step, which allows to access the version as follows:

version = readMavenPom.getVersion()

Ashlar

Thanks, great pointer to accomplish the end goal, but I was generally curious about the behavior of pipes in sh steps. Took me a minute to think about the sub-shelling and not a gotcha that I saw mentioned in the docs or occurred to me immediately when trying to do quick and dirty sh steps. – Negris

@Negris did you ever figure out more on the potential sub-shelling or an explanation behind that behavior? – Resh

not really, your example is pretty complex, not sure if it's because of sub-shelling or some syntax weirdness – Ashlar

Answer 5

So nothing detailed above worked for me using the scripted Jenkinsfile syntax with Groovy. I was able to get it working, however. The type of quotations you use are important. In the example below, I am trying to fetch the latest git tag from GitHub.

...

stage("Get latest git tag") {
  if (env.CHANGE_BRANCH == 'master') {
    sh 'git fetch --tags'
    TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
    LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
    VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
    echo "VERSION_NUMBER: ${VERSION_NUMBER}"
    sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
  }
}
...

Notice how the shell execution to assign LATEST_TAG works as expected (assigning the variable to v2.1.0). If we were to try the same thing (with single quotes) to assign VERSION_NUMBER, it would NOT work - the pipe messes everything up. Instead, we wrap the script in double quotes.

The first echo prints VERSION_NUMBER: 2.1.0 but the second prints VERSION_NUMBER:. If you want VERSION_NUMBER to be available in the shell commands, you have to assign the output of the shell command to env.VERSION_NUMBER as shown below:

...

stage("Get latest git tag") {
  if (env.CHANGE_BRANCH == 'master') {
    sh 'git fetch --tags'
    TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
    LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
    env.VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
    echo "VERSION_NUMBER: ${VERSION_NUMBER}"
    sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
  }
}
...

The first echo prints VERSION_NUMBER: 2.1.0 and the second prints VERSION_NUMBER: 2.1.0.

Egide

Answer 6

I am also struggling with the usage of pipe inside my jenkins pipeline but as a side note, if you want a simple way to extract the version of a maven pom, here's a very clean one I found in another post and that I'm using :

stage('Preparation') {
 version = getVersion()
 print "version : " + version
}
def getVersion() {
  def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
  matcher ? matcher[0][1] : null
}

gives you :

[Pipeline] echo
releaseVersion : 0.1.24
[Pipeline] sh

Trescott

There's no need do such thing anymore. See my answer using the readMavenPom step. – Ashlar

Great ! I will use that immediately. Thanks – Trescott

Answer 7

Here is a way I found that works, notice 3 different "types" of quotes:

1. Double quotes.
2. Single quotes.
3. Backticks.

Here is my code:

sh " ssh -i "+list[i][4]+" "+backupUser+"@"+list[i][0]+" /bin/bash -c  '`(ps uax |grep \"java -jar couchDbBackup.jar\" | grep -v \"grep\" | awk \"{print \$2}\" | xargs kill -9) ||true`' true" 

Explanation:

1. sh " - start and end with double quotes
2. ssh -i - pass private key to ssh
3. "+list[i][4]+" - add a value of the pipeline variable to the string (no magic so far just use double quotes to "exit" string)
4. "+backupUser+"@"+list[i][0]+" - user and hostname same like 4
5. /bin/bash -c - calling bash to wrap my command
6. ' - open a single quote that will span the whole command (else step 7 will not work) this is used to force Jenkins to treat the inside as a string.
7. ` - opening backticks. Apparently, bash doesn't care and treats them as quotes.
8. (ps uax |grep \"java -jar couchDbBackup.jar\" | grep -v \"grep\" | awk \"{print \$2}\" | xargs kill -9) - my set of commands with double quotes escaped with / for parameters. Everything is surrounded by round brackets.Brackets are not necessary if you do not need step 9
9. ||true - since my commands don't always return success I'm adding "OR true" to keep bash and Jenkins from stopping the pipeline.
10. `' - close backticks and single quotes.
11. true" - since Jenkins will consider text inside backticks as empty I add the final "true" (so that Jenkins will see something was passed to bash -c) and close the whole string with double quotes.

Intra