Can't jcmd, jps or jstat cassandra process within the docker container

Question

    $ jcmd -l 
    418 sun.tools.jcmd.JCmd -l

    $ jstat -gcutil -t 10 250ms 1
    10 not found

I am aware of the bug in jdk related to attaching jstat as root to a process running as a different user.

Here, this docker container has one user root and as can be seen below from the ps command, cassandra is running under root.

 $ whoami
 root

I have tried to do the following: $ sudo -u root jcmd -l

Any help is appreciated.

Docker container is debian:jessie running java version: openjdk version "1.8.0_66-internal"

Here's the output of ps -ef:

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:40 ?        00:00:00 /bin/bash /run.sh
root        10     1 11 17:40 ?        00:02:25 java -ea -javaagent:/usr/share/c
root       375     0  0 17:49 ?        00:00:00 bash
root       451   375  0 18:00 ?        00:00:00 ps -ef

Aside: jstack successfully dumps out the stack traces of the threads.

Arlyne

Answer 1

I know at least two possible reasons why this can happen.

1. Java is run with -XX:+PerfDisableSharedMem option. This option helps sometimes to reduce JVM safepoint pauses, but it also makes JVM invisible to jps and jstat. This is a very likely case, because you are running Cassandra, and recent Cassandra has this option ON by default.
2. Java process has a different mount namespace, so that /tmp of Java process is not physically the same directory as /tmp of your shell. The directory /tmp/hsperfdata_root must be accessible in order to use jps or jstat. This is also a plausible reason since you are using docker containers.

Flinty

(2.) probably is not the case for me as java.io.tmpdir hasn't been explicitly set. Sorry been swamped. Will run cassandra without PerfDisablesSharedMem later this evening and mark this as the right answer if I can validate it. – Arlyne

@Arlyne java.io.tmpdir does not matter at all. I was talking about Linux namespaces. Containers often make use of namespaces to provide a kind of isolation between processes, including isolation of directory structure. – Flinty

This may be an entirely different thread but just to clarify ... would running readlink /proc/3528/ns/pid on the pid of my cassandra process and readlink /proc/3528/ns/pid on the pid of my shell where i am running jstat from and comparing the symlinks be sufficient to see if they are in the same namespace? – Arlyne

@Arlyne Yes, but mnt instead of pid. Check that readlink /proc/$$/ns/mnt and readlink /proc/<java_pid>/ns/mnt return the same. – Flinty

In my case it turned out to be case 1 of @Flinty suggestions. – Arlyne

Interesting case for me the user has no name (whoami: cannot find name for user ID xxxxx), so the JVM . doesn't create /tmp/hsperfdata_$USERNAME directory (and bool UsePerfData = true flag is enabled). – Acerose