Remote monitoring with visualvm and JMX

Question

I would like to monitor a remotely running java (spring boot) application with jvisualvm (or jconsole). When running locally, I can see the managed beans in both jvisualvm and jconsole. When running remotely I cannot connect. I tried it with several different java processes (e.g. with spring xd). Looking for answers here on SO and on Google did not help.

These are my JAVA_OPTS (on the remote host):

$ echo $JAVA_OPTS
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=192.168.59.99

Then I simply start the program as follows (this is for spring xd, but I experience the same problem with other java programs).

$ bin/xd/xd-singlenode

The server process seems to pick up the options:

$ ps -ef | grep single
vagrant  22938 19917 99 06:38 pts/2    00:00:03 /usr/lib/jvm/java-8- oracle/jre/bin/java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=192.168.59.99 -Dspring.application.name=admin -Dlogging.config=file:/home/vagrant/spring-xd-1.1.0.RELEASE/xd/config///xd-singlenode-logger.properties -Dxd.home=/home/vagrant/spring-xd-1.1.0.RELEASE/xd -Dspring.config.location=file:/home/vagrant/spring-xd-1.1.0.RELEASE/xd/config// -Dxd.config.home=file:/home/vagrant/spring-xd-1.1.0.RELEASE/xd/config// -Dspring.config.name=servers,application -Dxd.module.config.location=file:/home/vagrant/spring-xd-1.1.0.RELEASE/xd/config//modules/ -Dxd.module.config.name=modules -classpath (...)

The java version on the remote host (ubuntu linux vm) is:

$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

The java version on the local machine (Mac OS) is slightly different:

$ java -version    
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

In jvisualvm I add the remote connection as follows (tried both with ssl connection and without):

This is the error message jvisualvm gives me:

I can connect from the local host to the remote host with the command telnet 192.168.59.99:9010, when the remote process is running -- so this does not seem to be a firewall problem.

Any help is highly appreciated.

Adaxial

Could you please try with the following Java options : -"Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.59.99" and also in the Jconsole use service:jmx:rmi:///jndi/rmi://192.168.59.99:9010/jmxrmi – Psychedelic

You saved my day. This works. Can you post this as an answer, then I will gladly accept. – Adaxial

Glad to hear that it worked! Posted as answer. – Psychedelic

Answer 1

Please use the following JVM options :

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.59.99

In the Jconsole use the following to connect:

service:jmx:rmi:///jndi/rmi://192.168.59.99:9010/jmxrmi

Psychedelic

It seems like a bad idea to make this accessible without authentication? – Heavyweight

@JackZachTibbles pretty common, though, inside corporate firewalls – Duggan

BTW, providing credentials through -D option makes it clearly readable as soon as a user has access to running process options (ps command on linux for instance) which is often possible as soon as you're able to connect through ssh to your server... so ... not sure if it brings a lot of security on the table – Catabasis

OK, so what's the difference from the options or connection that the author provided? Please show what was added/removed, without it is hard to make this fix for others. – Questa

Did not work for me because this answer does not mention that the connection needs a second (by default dynamically allocated) port to work. IMHO answer by thangdc94 should be the accepted one. – Mindexpanding

Answer 2

Arnab Biswas's anwser not work in my case. After an hour of researching, I found out that JMX runs on top of RMI, and as such, there are 2 ports that JMX utilizes:

• The JMX connect port. (-Dcom.sun.management.jmxremote.port)
• The (infamously) roaming RMI data port. (-Dcom.sun.management.jmxremote.rmi.port)

RMI data port will open a random port start from 1099. By setting the port used by the RMI registry and the RMI server to the same port, tunneling will be much easier.

So I need to add -Dcom.sun.management.jmxremote.rmi.port=9010 to JVM options

And I need to use the following JVM options :

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.59.99

Read more:

• http://hirt.se/blog/?p=289
• https://realjenius.com/2012/11/21/java7-jmx-tunneling-freedom/
• https://medium.com/codefountain/monitoring-using-java-visualvm-a25203d36390

Chemisorption

Thank you !!!! I was slowly going insane trying to get a remote connection to a firewalled host working... IMHO this should be the accepted answer. Seems like almost nobody tries to connect to a machine sitting behind a firewall so they all just suggest to set "com.sun.management.jmxremote.port" but nobody mentions "com.sun.management.jmxremote.rmi.port" ... – Mindexpanding

Ubuntu behind fire wall, UFW jmx, visualvm this should be done.. adding this comment so that it could pop up in google search :d – Garrulous

In addition to needing the rmi port declared, on my machine with Docker I noticed that if I try to forward a port different than the one declared in these JMX settings it will not work. For example JMX=8080 and Docker port set to map 8081:8080 will not work for accessing JMX. If the port is mapped as 8080:8080 it is fine. Many frustrating hours! – Arborescent

Answer 3

Let me discuss it in detail. I've just connected it to the remote server. U need to go to the folder where your jar file is. Suppose, you want to monitor X.jar file and suppose it is located in usr/local/ folder.. So, first cd into that folder. Then be sure which jdk is running. You can find it by executing command which java in linux. In the command line of the remote server (generally we use PuTTy cmd, I use MobaXterm). In the puTTy cmd write :

java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.10.0 -jar X.jar

Now let me explain in detail. Here, rmi port is needed to be defined because jmx uses rmi. If we don't define rmi port, it will automatically assign a random port. Then our connection won't be successful. We won't use authentication and ssl because this will cause difficulties. You can try this on your own if you want. In the hostname, you must specify your remote hostname which is the IP address. Now use visualvm located in your local computer. Generally, it can be found in this folder:

C://Program Files/Java/jdk1.8.0_201/bin 

If not then download it and paste it into the directory. Run visualvm from your local pc. You will see some options like local, remote. In the "remote" option select "Add Remote Host". Add your hostname. In my case, it was 192.168.10.0. Then right-click on your mouse and select "Add jmx connection". A window will open and insert your port number. In my case, it was 9010. You can specify any port number. Select "do not require ssl connection" and don't select the security credentials. Because we've added in the properties that authentication is false. Now connect and your connection will go smoothly. One more suggestion. You can connect using "jstatd connection. But it doesn't work. Doesn't show any output. Maybe because java 8 problems or so. For monitoring memory details you will need an updated version of "visualvm.exe". You can download it. You can connect using Authentication and SSL. The detailed steps using Authentication=true is discussed in this link: Connect to remote JMX Agent with authentication

The detailed steps using both authentication=true and SSL = true is discussed in this link: JConsole SSL/TLS with Password Authentication

Holmun