JMX and Debugging on Tomcat inside Docker

Question

I am trying to setup IntelliJ to connect to a Tomcat instance running in a Docker container. I would like to be able to use remote debugging and also deploy remotely using JMX.

I can enable remote debugging using the environment variables

JPDA_ADDRESS=8000
JPDA_TRANSPORT=dt_socket

and by starting Tomcat with catalina.sh jpda run, so remote debugging works without a problem.

I can also do this alternatively with

CATALINA_OPTS='-agentlib:jdwp=transport=dt_socket,address=8000,suspend=n,server=y'

and then I don't need to use catalina.sh jpda run

No matter what I do, I cannot get JMX to work. I verified that I have catalina-jmx-remote.jar in /usr/local/tomcat/lib`.

I have tried setting CATALINA_OPTS and JAVA_OPTS to

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=192.168.99.100
-Dcom.sun.management.jmxremote.ssl=false

I have verified that 192.168.99.100 is the IP of my docker machine. I have tried connecting to JMX with VisualJM and IntelliJ, it does not work. I have verified that the port 1099 is open and available from the host.

Tomcat is receiving the JMX args

20-Apr-2016 23:50:14.019 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=192.168.99.100 -Dcom.sun.management.jmxremote.ssl=false

Why can't I get JMX to work? There is no information available in any logs and this will not work no matter what I try.

Edit: lsof -i :1099 shows nothing running on that port

I am running on Mac OS X. It is a docker-machine but I believe docker uses virualbox on mac because it can't run containers natively. I have mapped the port. docker ps shows 0.0.0.0:1099->1099/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp. Ports 8080 and 8000 work so 1099 should be mapped correctly too.

Slimy

If you ssh onto the docker container and run hostname -i (assuming that's possible in your container), what is the result? – Winder

@EngineerDollery The result is 172.17.0.3 but I assumed that I was supposed to use the IP of the machine not the container. When I ping 192.168.99.100 (the IP of the docker machine), I get a response. When I ping 172.17.0.3, 100% packet loss – Slimy

Have you read this?: #31258468 or this?: #29958921 – Winder

@EngineerDollery I have. My situation is different. I am trying to access the docker JMX from the same machine the container is running on. As you can see from my edit, I see nothing running on port 1099 on the docker container so it looks like JMX isn't even running but I see no logs to find a reason why. – Slimy

Have you mapped the port to your host? What OS is your host? Are you using virtualbox or docker-machine? – Winder

@EngineerDollery I have appended my answer to the question. Also, I think this is irrelevant considering that in the docker container, there are no processes operating on 1099. – Slimy

Do you still have this problem? I think this may be related to using the catalina-jmx-remote.jar and the JmxRemoteLifecycleListener together with the com.sun.management.jmxremote.* system properties. From my understanding the JmxRemoteLifecycleListener replaces the port system properties (see here). Not sure how/if it's is supposed to work of you specifiy both... – Chemosmosis

See here for a working setup without the JmxRemoteLifecycleListener. Otherwise could you add the configuration of the JmxRemoteLifecycleListener to your question as well? – Chemosmosis

Answer 1

I was able to connect when I used 0.0.0.0 for jmxremote.host and server.hostname

 HOST=0.0.0.0
    java -Xmn100M  -XX:+PrintGCDetails  -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -Xmx384M $JAVA_OPTS\
     -Dcom.sun.management.config.file=/opt/app/management.properties \
     -Djava.util.logging.config.file=/opt/app/logging.properties \
     -Dcom.sun.management.jmxremote.port=$JMX_PORT \
     -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT \
     -Dcom.sun.management.jmxremote.host=$HOST \
     -Djava.rmi.server.hostname=$HOST \
     -jar /opt/app/app.jar

Irritation

Finally! I wish I have found it earlier and not after spending my entire day trying to make darn thing work. Thanks so much. – Highlight