Access JMX inside docker container with mapped ports

Question

I am trying to access an application running inside a docker container with JMX.

This is similar to this question and that solution works when the ports inside the docker image are mapped to the same ports outside the image. However, I sometimes want to map the port to a different port.

I am setting these properties in the managed application.

-Dcom.sun.management.jmxremote.port=9832 
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.rmi.port=9832 
-Djava.rmi.server.hostname=192.168.99.100 
-Djava.rmi.server.logCalls=true

This works fine when the docker container maps port 9832 to 9832. I can connect via JConsole or our own application. If instead the port is mapped to another port, then I am unable to access the application from JConsole or our application.

I suspected that one or two of the port numbers needed to be the external port (just like java.rmi.server.hostname is the external address, not the internal one). However, it fails with all four combinations of port numbers.

Two of the combinations produce no log output from the server. One (I forget which) produces this output:

Feb 09, 2016 10:35:54 PM org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl start
INFO: AMQ221001: Apache ActiveMQ Artemis Message Broker version 1.1.0 [nodeID=7a6e038e-cf7d-11e5-b566-31dc437b2d1a] 
HTTP Server started at http://0.0.0.0:8161
Feb 09, 2016 10:36:06 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(1)-192.168.99.1: [192.168.99.1: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease)]
Feb 09, 2016 10:36:08 PM sun.rmi.transport.Transport serviceCall
FINE: RMI TCP Connection(1)-192.168.99.1: [192.168.99.1] exception: 
java.rmi.NoSuchObjectException: no such object in table
    at sun.rmi.transport.Transport.serviceCall(Transport.java:177)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Feb 09, 2016 10:40:24 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(2)-192.168.99.1: [192.168.99.1: sun.rmi.transport.DGCImpl[0:0:0, 2]: void clean(java.rmi.server.ObjID[], long, java.rmi.dgc.VMID, boolean)]

The other produces this output.

HTTP Server started at http://0.0.0.0:8161
Feb 09, 2016 10:14:13 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(1)-192.168.99.1: [192.168.99.1: sun.management.jmxremote.SingleEntryRegistry[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 09, 2016 10:14:17 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(1)-192.168.99.1: [192.168.99.1: sun.management.jmxremote.SingleEntryRegistry[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 09, 2016 10:14:17 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(1)-192.168.99.1: [192.168.99.1: sun.management.jmxremote.SingleEntryRegistry[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]

Given these failures, I suspect that one or both of the port properties are used in such a way that they sometimes need to be the internal port and sometimes need to be the external port. That would mean that JMX access is not possible when the ports are mapped to a different location.

I can access the mapped port via telnet 192.168.99.100 <mapped port>, so I know that the mapping is working.

Gyrate

Did you check, during no logs time, that firewall is not blocking the port exposed by docker container ? – Vick

@TyagiAkhilesh I can connect to the port with telnet, so the port is accessible. – Gyrate

From memory JMX tells the client what the second port to connect to is on the initial connection. So the client expects to be able to connect to the app configured port. Just use the mapped ports. If you really you want to use different NAT ports, use a socks proxy. – Natasha

We need more likes for this question. – Tripletail

Have you tried this with 2 different ports and different mappings? – Mongoose

Answer 1

I had the same problem and over almost three days in a row, I've just given up and I did a solution that is not great but it worked. So, my solution was: instead of setting a fixed port for JMX connection and map it using a different external port, I've created an environment variable called JMX_MNG_PORT on my docker-compose.yml.

...
ports:
  - '9011:9011'
...
environment:
  - JMX_MNG_PORT=9011
...

Then I configured on my docker-entrypoint.sh, the JAVA_OPTS variable to receive the JMX_MNG_PORT.

...
JAVA_OPTS=$JAVA_OPTS -Dcom.sun.management.jmxremote.port=${JMX_MNG_PORT}
JAVA_OPTS=$JAVA_OPTS -Dcom.sun.management.jmxremote.rmi.port=${JMX_MNG_PORT}
...

Now, on each container being configured (docker-compose.yml), I just have to configure the JMX_MNG_PORT and expose the port using THE SAME NUMBER.

REMEMBER, I don't know why, but if you use another port then the one configured on jmxremote.port and jmxremote.rmi.port, it will not work.

I hope it solves the case for others like me.

Trophoplasm