VisualVM through firewalls - RMI troubleshooting

Question

Sorry for this question which must have been asked many times, but I can't succeed in resolving my problem. I've read a lot of blogs, sites, forums, .... and didn't find any solution in my case.

Case : I need to connect VisualVM on my box to distant servers (tomcats, weblogics) for performance / threads / memory monitoring. Those servers are installed on (physical or virtual) machines which are protected by a firewall. Large intervals of ports are open in the firewall and can be used, but not all ports.

Tests

• I've tried direct connections through JMX in VisualVM, using following JVM options on the server-side at server startup :

  -Djava.rmi.server.hostname=[hostname]
  -Dcom.sun.management.jmxremote
  -Dcom.sun.management.jmxremote.port=[port]
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.authenticate=false
  

I've precised the hostname because from my network the hostname and the IP address of the server are not the same than those from the network of the remote server.

No success, VisualVM always seems searching for an unknown server.

• tried starting jstatd on the server-side on a port accessible (-p option) from my box (telnet on this port works), but when launching visualVM on this host with the jstatd port, it still seems waiting for something unreachable.... Same behavior with jps connecting to this remote host.

• tried using the same tools on a server with less network protection, and it works. So I have seen the connections between my box and the server and they are done on ports different from what I've specified to jstatd. I understand that this port is needed for first communication (kind of handshake) and real communications are done on other ports, but not predictible (ex: 60305, 55197, ...). Not sure I understand very well how RMI works.

Please, help me, I'm going crazy !

Geri

If you are on Java 7 update 4 there is hope with the flag `` -Dcom.sun.management.jmxremote.rmi.port=7091`` See this blog post: hirt.se/blog/?p=289 – Dizzy

Answer 1

Unfortunately JMX tries to open ports other than the one you configure. Just yesterday I succeeded connecting to tomcat behind firewall via JMX. The two tricky parts are:

• put a file called jmxremote.access in CATALINA_HOME/conf, which contains the following lines:

  monitorRole readonly
  controlRole readwrite
  

• in server.xml set the ports that will be used by jmx, via a special tomcat listener (catalina-jmx-remote.jar required in /lib):

  <Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" 
      rmiRegistryPortPlatform="9009" rmiServerPortPlatform="9010" />
  

Then open these two ports on the firewall. It works. But that's just for tomcat.

Another option is to use ssh tunnelling. In short - you connect via SSH and configure it to forward some local port (where the jmx client is running) to some ports on the other side of the tunnel.

References:

• Connecting to JMX on Tomcat 6 through a firewall
• Connecting MySQL server on Amazon EC2 instance using ssh tunnel

Endosmosis

You should add, that one also has to copy the catalina-jmx-remote.jar into CATALINA_HOME/libs otherwise a ClassNotFoundException is thrown. For further reading on this topic see tomcat.apache.org/tomcat-6.0-doc/config/… – Dinny

Answer 2

Here are the steps to do this:

1. Launch an ejstatd in your remote host this way (in ejstatd folder): mvn exec:java -Djava.rmi.server.hostname=[remote_host_name] -Dexec.args="-pr 1099 -ph 1100 -pv 1101" (used for "jstatd" type connection)
2. Launch your Java application with those additional Java parameters: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1102 -Dcom.sun.management.jmxremote.rmi.port=1102 -Djava.rmi.server.hostname=[remote_host_name] (used for "JMX" type connection) (java.rmi.server.hostname is required here only because the IP and hostname from your network is not the same as the server point of view)
3. Open those 4 ports on your remote host and make them available to your local machine: 1099, 1100, 1101 and 1102
4. Launch JVisualVM
   1. Right-click on "Remote" > "Add Remote Host..." and enter your remote host name in "Host name" (if you don't use the port 1099, you can change this in the "Advanced Settings")
   2. Right-click on the remote host you've just created > "Add JMX Connection..." and enter "[remote_host_name]:1102" in "Connection" input, and check "Do not require SSL connection"
   3. Your Java process will appear twice: one from the "jstatd" connection type, and one from the "JMX" connection type.

Disclaimer: I'm the author of the open source ejstatd tool.

Malvern

Answer 3

On your [hostname], open up [port] and tcp port range 40000-60000 for your IP only. This did the trick for me fairly well.

Mancy

Why has this answer been downvoted so much? This is the only one that actually helped me. – Periodontics

Because just opening nearly all ports of your machine to the rest of the world might work on a dev machine but for sure is not feasible in a production/enterprise environment. – Cleasta