Debugging JConsole Connection Failed

Question

I have a web application deployed to a remote resin server, and it has JMX turned on.

I can telnet to the remote server i.e

franz@see:/tmp$ telnet <remote-ip> 5555
Trying <remote-ip>...
Connected to <remote-ip>.
Escape character is '^]'.
��sr5javax.management.remote.message.HandshakeBeginMessage�,���6profilestLjava/lang/String;Lversionq~xppt1.0^]

telnet> q
Connection closed.

But I cannot connect to it using my JConsole

$JAVA_HOME/bin/java -cp $JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:pm-common/lib/jmxremote_optional-1_0_1_3.jar sun.tools.jconsole.JConsole service:jmx:jmxmp://<remote-ip>:5555

I have tried this with the following java versions but I get a 'Connection Failed' on both instances.

## where JAVA_HOME=/opt/java/64/jdk1.5.0_22
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)

## where JAVA_HOME=/opt/java/64/jdk1.6.0_17
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)

Do you guys have any idea as to how to debug this (i.e. find out what's wrong)?

Roofdeck

Answer 1

Make sure you are running your application with following java properties set

-Dcom.sun.management.jmxremote.port=9005
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Try to connect now. If you want to debug this ,you can run the jconsole with following command

jconsole -J-Djava.util.logging.config.file=path_to_logging.properties_for_jconsole

Below is the content of logging.properties file

Logging.properties

handlers = java.util.logging.ConsoleHandler


.level = INFO

java.util.logging.ConsoleHandler.level = FINEST

java.util.logging.ConsoleHandler.formatter = \

java.util.logging.SimpleFormatter

// Use FINER or FINEST for javax.management.remote.level - FINEST is

// very verbose...

javax.management.level = FINEST

javax.management.remote.level = FINER

Once you run jconsole a separate window will pop up displaying logs.

Keeney

Ey! what about the missing property 'java.rmi.server.hostname'. look out for this post : gubatron.com/blog/2010/11/21/… !! Hope it helps! – Fret

Also make sure that the password file is read protected! ie. only the user can read it - do this with chmod 600 – Haldis

Why does everyone keep going on about remote connections? I'm trying to connect locally and it still doesn't work. "Connection Failed: Retry? The connection to 15663 did not succeed." WHY? Where's the logs? Where's the error? Such a joke. – Aileen

I needed to access the jmx port through ssl tunneling. For me it was neccessary to add "java.rmi.server.hostname" property, and have to set it to 127.0.0.1, not to the real IP of the machine! As soon as I set it to 127.0.0.1, it started working via the tunnel... – Succinate

Answer 2

if you run jconsole -debug it gives you more diagnostic info on the failure. See the Daniel Fuchs blog entry "Troubleshooting connection problems in JConsole".

I did this and it showed me I was using 32 bit jconsole the target process was started with a different (64 bit) jvm, so apparently this isn't allowed and it was thus failing.

Fourteenth

Answer 3

This finally made it work for me : Giving this extra option: -Djava.rmi.server.hostname=<ip addres where jvm is running

So all the vm arguments used to open jconsole from a remote machine, the jvm on the remote machine must be started with

 -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.port=<port> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<ip address>

The entire process is listed here

Alpenstock

This worked for me as well. Seems when you have NAT going on, you need to specify the server address. – Properly

After long search, adding -Djava.rmi.server.hostname=<ip address> finally works for me when connecting from different machine. Below is the full arguments: java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false \ -Dcom.sun.management.jmxremote.port=9999 \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Djava.rmi.server.hostname=<IP> \ com.example.Main – Ahner

Answer 4

I encountered the same Problem when starting the Java-Process through cygwin. JConsole cannot connect. Started it through win7-cmd everything works as expected.

Altamira

Answer 5

I don't know if this is helpful, but perhaps you should use the the jconsole binary in the JDK bin directory rather than using the undocumented (and subject to change) sun.* classes to start up the console

Photochromy

Still did not work. And I don't know how to debug this either. – Roofdeck

This article (blogs.sun.com/jmxetc/entry/…) explaining common connection problems may be useful. If nothign works please review/post debug logs – Photochromy

Answer 6

If your application is running on JDK 1.6 then you should be able to connect it. If it is using JDK prior to 1.6 then run it with specifying the following JVM argument

-Dcom.sun.management.jmxremote

Deathlike

Does not work (tried both on JDK1.5 & JDK1.6). And I don't know how to debug it either. – Roofdeck

Answer 7

I was having similar issue that remote machine was behind firewall and firewall was blocking ports defined by -Dcom.sun.management.jmxremote.port and RMI 46924. After allowing me to cnnect to these port I connected succesfully.

Garrott

In my case RMI port was 43702 and I found that out by debugging with logging.properties – Samson

Answer 8

If you are accessing a machine behind a firewall, you need to open both JMX and RMI ports.
In this context, you are much better off forcing the value for RMI than relying on the auto assigned
In my case, I was trying to access Tomcat so I had to do the following:

#!/bin/sh
CATALINA_OPTS="$CATALINA_OPTS -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8008 -Dcom.sun.management.jmxremote.rmi.port=8007 -Dcom.sun.
management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

and then

firewall-cmd --zone=public --add-port=8008/tcp --permanent
firewall-cmd --zone=public --add-port=8007/tcp --permanent
firewall-cmd --reload

Ramshackle

Answer 9

For me, I was using a local process connection and had the same issue as OP. The misconception I had was, "Connection can't be refused if it local, right?" Not necessarily.

Another user, JeneralJames, mentioned the exact same symptoms I had in a comment on samarth's answer.

What resolved it was disconnecting from VPN (or alternatively enabling LAN access while connected to VPN - a non-default configuration for my VPN settings).

I discovered the issue by running jconsole -debug as suggested by rogerdpack, and it showed me an IP I was not expecting in the connection attempt (despite being a connection to a local process). My VPN settings at the time disallowed connections on the local network entirely, so java was looking up the local IP, getting the value, but that IP is not allowed to connect due to the VPN configuration, so the "local connection" was rejected.

Apparently jconsole (or any other MBeans using JMX app for that matter) will use this incorrect IP by default even for local connections. I hope this helps someone else out, I have spent 3 days perusing multiple stackoverflow threads and java documentation to no avail until discovering that -debug option.

Sisile