Can not take jstat metrics using bash as Sensu plugin

Question

I have created bash scirpt that takes jstat metrics of my jvm instances!

Here is the output example :

demo.server1.sms.jstat.eden 24.34   0
demo.server1.lcms.jstat.eden    54.92   0
demo.server1.lms.jstat.eden 89.49   0
demo.server1.tms.jstat.eden 86.05   0

But when the Sensu-client runs this script it returns

Could not attach to 8584
Could not attach to 8588
Could not attach to 17141
Could not attach to 8628
demo.server1.sms.jstat.eden     0
demo.server1.lcms.jstat.eden    0
demo.server1.lms.jstat.eden     0
demo.server1.tms.jstat.eden     0

Here is the example of check_cron.json

{
  "checks": {
    "jstat_metrics": {
      "type": "metric",
      "handlers": ["graphite"],
      "command": "/etc/sensu/plugins/jstat-metrics.sh",
      "interval": 5,
          "subscribers": [ "webservers" ]
    }
  }
}

And piece of my bash script

jvm_list=("sms:$sms" "lcms:$lcms" "lms:$lms" "tms:$tms" "ums:$ums")
for jvm_instance in ${jvm_list[@]}; do
    project=${jvm_instance%%:*}
    pid=${jvm_instance#*:}
        if [ "$pid" ]; then
          metric=`jstat -gc $pid|tail -n 1`
          output=$output$'\n'"demo.server1.$project.jstat.eden"$'\t'`echo $metric |awk '{ print $3}'`$'\t0'
        fi
done
echo "$output"

I find out that problem is with jstat and i tried to write full jstat path like /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/jstat -gc $pid|tail -n 1 but it didn't help!

By the way if i will comment this row the output like "Could not attach to 8584" disappears!

Liddle

Answer 1

Yes you're right sensu runs all script as sensu user. To use jstat you have to add sensu to a sudoers.

just add file /etc/sudoers.d/sensu

Example:

Defaults:sensu !requiretty

Defaults:sensu secure_path = /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

sensu ALL = NOPASSWD: /etc/sensu/plugins/jsat-metrics.rb

Liddle

Answer 2

I'm not a Java or Sensu user, but I can guess what happens.

Most likely, sensu-client runs your script as a user different from the one you use when testing manually, which doesn't have permissions to "attach" (whatever that means) to your jvm instances.

To verify this you can add invocation of "whoami" to your script, run it from sensu-client again, see what user it runs your script under and, if it is different, try to run your script as that user.

Seaman

You were right the script was running under sensu user (it was surprise for me )!!! So i had to use jstatd !!! it's a deamon that can let other user's to take jstat metrics remotely ! – Liddle

Great! Glad to hear that :) – Seaman

jstat is best run as root as you found out....any other user likely will be unable to connect to the process. – Impend

Not the answer of the question but since Sensu user doesnt have shell, I use sudo to run a command by that user for testing, this is useful to catch the issues beforehand. $ sudo -H -u sensu bash -c '/opt/sensu/embedded/bin/check-disk-usage.rb' CheckDisk OK: All disk usage under 85% and inode usage under 85% – Perspiration