Unable to remotely connect to JMX?

Question

For some weird reason I am not able to connect using VisualVM or jconsole to a JMX.

The parameters used to start the VM to be monitored:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=1100

I checked, and I can telnet to this port, from both locally and remotely.

Still, VisualVM or jconsole are failing to connect, after spending some considerably time trying to.

REMOTE MACHINE with JMX (debian)
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

MY WORKSTATION (OS X)
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

What is the problem?

Don

After supplying only -Dcom.sun.management.jmxremote.port=1100, can you run VisualVM locally, but connect via the port instead of through attach? – Rocher

If there is a firewall in play see [this issue.][1] [1]: https://mcmap.net/q/173204/-has-anyone-ever-got-a-remote-jmx-jconsole-to-work – Reckoner

Much Much easier to set up that through ssh tunnelling – Baumgardner

@Don , I have tried all the suggested answers below still facing cannot connect to remote while trying to connect a remote ec2 process from local. Could you please help ? – Shishko

On weblogic10.3.4 (maybe other versions as well), you MUST use the weblogic console's username/password for authentication EVEN IF you've added the java start argument: -Dcom.sun.management.jmxremote.authenticate=false (!!!) – Crewel

Answer 1

Add -Djava.rmi.server.hostname = host ip. Even i faced the same problem and this did the trick.

Addition of this -Djava.rmi.server.hostname = host ip forces RMI service to use the host ip instead of 127.0.0.1

Niacin

I can confirm that adding this setting solved my problem with connecting from VisualVM to a remote VM. Nice answer. – Botswana

I did a TCP dump on the communication when JConsole tries to connect. Apparently in the handshake with the JMX port, it relays back the host given at -Djava.rmi.server.hostname and then it tries to connect to it. – Seaway

Where to add suggests variable change ? – Meltage

This answer helped me to connect to a JVM running in Kubernetes via a port forwarding: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10099 -Dcom.sun.management.jmxremote.rmi.port=10099 -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false I.e. setting the hostname to 127.0.0.1 and setting both jmx and rmi port to the same value! – Ambriz

Answer 2

These are the steps that worked for me (Debian behind firewall on the server side was reached over VPN from my local Mac):

1. Check server public ip

   ifconfig

2. Use JVM params:

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=[jmx port]
    -Dcom.sun.management.jmxremote.local.only=false
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false
    -Djava.rmi.server.hostname=[server ip from step 1]

3. Run application

4. Find process ID of the running java process

5. Check all ports used by JMX/RMI

   netstat -lp | grep [pid from step 4]

6. Open all ports from step 5 on the firewall

Voila.

Trespass

Thanks! I was missing netstat -lp | grep [pid from step 4] – Botulism

In my case, I was missing the "java.rmi.server.hostname" ... after, adding that property, I was able to connect to my remote VM usign java VisualVM – Nasia

In our case, turned out it more ports were required than just the one specified via jmxremote.port. We asked ops to open all ports to the machine behind VPN and were able to connect. Unexpected and have not reduced port exposure the machine yet to give more data. – Naturopathy

@Trespass port opening to be done in remote machine or local ? – Shishko

@Shishko - remote machine – Trespass

@Trespass ... Still not resolved, please help me here stackoverflow.com/q/48516329/3301316 – Shishko

ifconfig is better suitable for knowing the server public ip instead of hostname -i – Dittography

Answer 3

In addition to listening to the port you specified (1100) the JMX server also listens to a randomly chosen (ephemeral) port. Check, e.g. with lsof -i|grep java if you are on linux/osx, which ports the java process listens to and make sure your firewall is open for the ephemeral port as well.

Guib

This is on intranet and there is no firewall, in addition I even tried to connect locally using the DHCP IP and it fails. Interesting but VisualVM is able to detect local instances and connect to them, but probably it does not use TCP ports for these. – Don

The VM parameters looks correct. Wrong options/url to jconsole? – Guib

On VisualVM I right clicked on remote add JMX Connection and introduced subdomain.example.com:1100. – Don

I also tried the second syntax service:jmx:rmi:///jndi/rmi://subdomain.example.com:1100/jmxrmi but the results were the same: a long, long delay of "connecting to service" that fails, probably with a timeout. – Don

Have you tried using -Djava.rmi.server.hostname=subdomain.example.com, could be that the RMI service uses localhost. – Guib

Answer 4

I experienced the problem where it said 'Adding ' forever and didn't seem to be able to connect. I got passed the problem by changing the jvisualvm proxy settings (Tools->options->network). Once I changed the option to No Proxy, I was able to connect. My jvm was started with the following options:

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=2222 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=<external_IP_of_server> 

Then when I added the jmx connection, I specified "external_IP_of_server:2222"

Hazing

Answer 5

I had a similar issue when using port forwarding. I have a remote machine with Tomcat listening for JMX interactions on localhost:9000.

From my local machine, I'm used to do port-forwarding with:

ssh -L 9001:localhost:9000 tomcat.example.com

(so remote port 9000 is forwarded to my local machine's port 9001).

Then when I tried to use VisualVM to connect to localhost:9001, the connection was refused. JMX seems to require port numbers on both sides to be identical.

So my solution was using port numbers 9000 and 9000:

    ssh -L 9000:localhost:9000 tomcat.example.com

Now my local machine's VisualVM connects successfully to the remote machine's Tomcat via localhost:9000.

Make sure that you don't have any other service (Tomcat on dev machine?) listening on the same port.

Also take a look at setting up parameters correctly.

Carbajal

Answer 6

My two cents to the above answers..

I see most of the answers has mentioned only about the hostnames but no port. If we haven't specified ports, then server will dynamically assign RMI port. There wont be any issues, if both the servers are in same subnet or no firewall issues. If there is any concerns, we can add below JVM parameter to freeze.

-Dcom.sun.management.jmxremote.rmi.port

Ex:

<option name="-Dcom.sun.management.jmxremote.rmi.port" value="11001"/>

Make sure both RMI and JMX ports should be the same. For more, click here

Intermingle

Thanks for that answer. I already spend hours to find that hint. Now everything works for me: – Emotionalize

This was also the solution for us! thanks! – Validate

Thanks, this worked for me. I was only specifying -Dcom.sun.management.jmxremote.port and it was not working. working after adding -Dcom.sun.management.jmxremote.rmi.port – Repast

Answer 7

Since I just joined I can't upvote Hett's answer, but it saved my life from another week of trial and error!

This is an example of a working Dockerfile:

FROM store/oracle/serverjre:8 

RUN mkdir -p /opt/app

ENV APP_PATH /opt/app

WORKDIR $APP_PATH   

COPY . $APP_PATH

CMD ["java", \
     "-Dcom.sun.management.jmxremote", \
     "-Dcom.sun.management.jmxremote.port=9010", \
     "-Dcom.sun.management.jmxremote.rmi.port=9010", \
     "-Dcom.sun.management.jmxremote.authenticate=false", \
     "-Dcom.sun.management.jmxremote.ssl=false", \
     "-Djava.rmi.server.hostname=12.345.67.89", \
     "-jar", \
     "app-service-0.0.1-SNAPSHOT.jar"]

EXPOSE 9010

Trilateration

Answer 8

If you run jar file (via -jar option), you must specifie all other jvm options before -jar option!

Aegean

Answer 9

I found the problem, my rmi service was running on the host ip which was "127.0.0.1". To connect remotely to the jvm I had to bind the external ip to the hostname. To do this in unix systems use command hostname to get the name of the hostname. Then check the ip that is assigned to the hostname, to find out this use ping $(hostname) you will see that system is pinging the hosname's ip. If your host ip was the default "127.0.0.1" and you wanted to change it, just edit the file /etc/hosts as superuser. After rebooting the rmi service, you can reach it from the remote machine.

Underlayer

hostname will give you the wrong answer in some environments. To get the correct external IP on AWS EC2 use: ip=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4) – Henriques

Answer 10

look in /etc/hosts if you don't have a wrong IP for your machine example : 127.0.0.1 localhost 127.0.0.2 your_machine 185.12.58.2 your_machine (the good IP for your machine)

JMX take the IP 127.0.0.2 and forget the other

Fibrilliform

Answer 11

The following worked for me, thanks to @Arpit Agarwal. Added this additional jvm parameter which worked for me.

    -Djava.rmi.server.hostname=192.168.1.16

Complete list which worked for me.

 -Dcom.sun.management.jmxremote 
 -Dcom.sun.management.jmxremote.port=21845
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.local.only=false 
 -Djava.rmi.server.hostname=192.168.1.16
 -Dcom.sun.management.jmxremote.rmi.port=10099 

Parent