Java jconsole jmx connection failure

Question

I'm trying to connect jconsole to a jvm invoked by:

java \
-Djava.util.logging.config.file=./logging.properties \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=12700 \
-cp . Sleep

I then try to start jconsole with:

jconsole -J-Djava.util.logging.config.file=./logging.properties

The loggin.properties file includes:

sun.rmi.level=FINEST

After opening a socket to port 12700, rmi then appears to attempt a connection on another port :

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.40.243.12:12700/jmxrmi] connecting...
Jan 5, 2012 2:30:42 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.40.243.12:12700/jmxrmi] finding stub...
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint <clinit>
FINE: JConsole.addHost: localHostKnown = true, localHost = 10.206.6.59
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPTransport <init>
FINE: JConsole.addHost: Version = 2, ep = [10.206.6.59:0]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
FINE: JConsole.addHost: created local endpoint for socket factory null on port 0
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [10.40.243.12:12700]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: 10.40.243.12, port: 12700
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: server suggested 10.206.6.59:12306
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: using 10.206.6.59:0
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINER: JConsole.addHost: create call context
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef logClientCall
FINER: JConsole.addHost: outbound call: [endpoint:[10.40.243.12:12700](remote),objID:[0:0:0, 0]] : sun.rmi.registry.RegistryImpl_Stub[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall <init>
FINER: JConsole.addHost: write remote call header...
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall getOutputStream
FINER: JConsole.addHost: getting output stream
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef invoke
FINER: JConsole.addHost: execute call
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall getInputStream
FINER: JConsole.addHost: getting input stream
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "javax.management.remote.rmi.RMIServerImpl_Stub", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "javax.management.remote.rmi.RMIServerImpl_Stub" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "java.rmi.server.RemoteStub", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "java.rmi.server.RemoteStub" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "java.rmi.server.RemoteObject", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "java.rmi.server.RemoteObject" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef done
FINE: JConsole.addHost: free connection (reuse = true)
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: reuse connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: create reaper
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [tdiap12:41096]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: tdiap12, port: 41096

The first connection attempt succeeds because on the remote server, I can see ESTABLISHED connection.

wse2tst@tdiap12:~> netstat -a | grep 12700
tcp        0      0 *:12700                     *:*                         LISTEN      
tcp        0      0 tdiap12.vgcar.net:12700     per-00c0016253a2.vgca:12252 ESTABLISHED 

The packet trace indicates a successful connection as well.

The second connection to port 41096 times out, and the jconsole application reports a "Connection failure" and terminates the first connection as well. Why is a second connection being attempted? Is there any way to specify the port for this second connection? The target server is tightly controlled, and other ports are blocked by firewall rules. On multiple connection attempts, the second port will change to different random values.

Thanks for any help, Steve

Worm

are you sure no firewalls on your machine blocking 12700? I suspect because of some blocking, jmx trying alternative port. Try by disabling your firewall. – Misty

When I try to connect from my local machine, the remote machine shows the following:wse2tst@tdiap12:~> netstat -a | grep 12700 tcp 0 0 *:12700 *:* LISTEN tcp 0 0 tdiap12.vgcar.net:12700 per-00c0016253a2.vgca:12252 ESTABLISHED I'm assuming the connection is being established and is not being blocked by a local firewall rule. The packet trace also indicates that the connection is successful. – Worm

I guess you need to add -Dcom.sun.management.jmxremote while bringing up the jvm you specified on top of this question, don't we need to? – Misty

I haven't seen that in the documentation. What would that do? The first two -Dcom.sun.management.jmxremote values turn of encryption and security. The third defines the listening port for incoming JMX connections. I don't think setting -Dcom.sun.management.jmxremote by itself would have any affect. – Worm

Try it. It let your JVM look for remote connections. – Misty

Added -Dcom.sun.management.jmxremote \\ to the command line. No difference in behavior. This time it tried connecting to port 53657. It's different every time. – Worm

Then I am sorry! As final try, I would try doing same thing from another remote machine (if available). I strongly suspect there is something wrong with remote machine you are trying to connect. – Misty

@Worm , Cannot connect to XXXXXXXX:8091 using service:jmx:rmi:///jndi/rmi://XXXXXX:8091/jmxrmi I am getting this error, where I am going wrong ? – Moulin

Answer 1

The out-of-the-box JMX implementation uses two ports - one for the registry (the one that you specified) and one for the actual connection which is selected randomly (!). This is quite a design flaw since the random selection of the second port makes it very hard to configure a firewall.

There are ways around it however - either you can do it yourself manually, or if you are using Tomcat you let it handle it for you.

Shriner

Turns out another developer had already coded a custom agent to get around the issue of the firewall blocking the second random port. I was determined to get to the bottom of the problem instead of having to install the custom agent everywhere I needed to monitor. (I have to monitor tons of application servers.) I had gotten extremely frustrated because I never found the documentation identifying the second port requirement, but you at least confirmed my observations, and that's good enough for me. Thanks for the answer. – Worm

Answer 2

new and better solution than using a manually coded agent, is to use the new parameter which seems to be introduced in Java 7:

-Dcom.sun.management.jmxremote.rmi.port=7091

So in combination - same port can be used:

-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.rmi.port=7091

Solution found here: http://hirt.se/blog/?p=289

Forensic

Answer 3

make sure you run your application using below command,

java -Dcom.sun.management.jmxremote.port=9595 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -jar <your jar file name>

then open jconsole with admin privilege; you might be able to connect without any issue.

if you want to run multiple jar files, make sure to use different ports; otherwise it will give error saying it's already bind by the JVM

Supportable

Please don't post duplicate answers to multiple questions. – Groundhog

Answer 4

You can set the second port with parameter -"Dcom.sun.management.jmxremote.rmi.port" where port number can be same as that of first one . Even after setting this it will promt for insecure connection ... please allow insecure connection then only it will allow you to proceed further otherwise till then in log you will find error failed to connect: java.lang.SecurityException: Expecting a sun.rmi.server.UnicastRef2 remote reference in stub!

Thanks

Stringpiece