Connect JConsole to GlassFish server running on Docker behind a NAT server

Question

My infrastructure is architected this way (on AWS):

|NAT| <--> |ServerA| <--> |DockerContainer| <--> |GlassFish4.0|

ServerA has a Docker container running Glassfish version 4.0

ServerA has internet access thru NAT server.

To connect to ServerA I've to SSH into the NAT server and then SSH to ServerA.

This is running smoothly with no problems at all.

Now I need to connect JConsole on my local machine to the GlassFish server and here is where I've problems and need some help.

I added the following JVM options to the domain.xml file:

<jvm-options>-Dsun.management.jmxremote.level=FINEST</jvm-options>
<jvm-options>-Dsun.management.jmxremote.handlers=java.util.logging.ConsoleHandler</jvm-options>
<jvm-options>-Djava.util.logging.ConsoleHandler.level=FINEST</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.local.only=false</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.ssl=false</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.authenticate=false</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.port=9010</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.rmi.port=9010</jvm-options>
<jvm-options>-Dcom.sun.management.jmxremote.host=0.0.0.0</jvm-options>
<jvm-options>-Djava.rmi.server.hostname=10.0.0.115</jvm-options>

Where 10.0.0.115 (for java.rmi.server.hostname) is the local address of ServerA.

Then I create a SSH tunnel from my local machine to the NAT server like this:

ssh -L 9010:localhost:9010 nat

And then I make another SSH tunnel to ServerA like this:

ssh -L 9010:localhost:9010 serverA

On the docker container I expose port 9010 on the docker run, like this:

docker run --restart=always --name $CONTAINER_NAME  \
-d  \
-p 4848:4848 \
-p 8080:8080 \
-p 9010:9010 \
my container

Now when I try to connect JConsole I use localhost:9010 but it fails.

I tried to telnet to localhost 9010 and looks like it succeeds:

telnet localhost 9010
Trying ::1...
Connected to localhost.
Escape character is '^]'.




^C^C^C^C^C^C


^]
::
exit
^Z^Z^X^C^V^M

Connection closed by foreign host.

After some time I get the Connection closed by foreign host.

I've tried using VisualJM but if I input localhost:9010 VisualJM considers it has "really" localhost and does not allow me to add a remote server and then JMX.

I've opened all traffic to NAT and ServerA on the security groups to my own IP and I got no success so I'm assuming it is not a firewall issue.

Can someone help me out to spot what am I doing wrong in here?

EDIT:

When using jconsole -debug on my local machine, after trying to connect to the docker JVM I get the following exception:

java.lang.SecurityException: Expecting a javax.rmi.ssl.SslRMIClientSocketFactory RMI client socket factory in stub!
    at javax.management.remote.rmi.RMIConnector.checkStub(RMIConnector.java:1903)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:295)
    at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:355)
    at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
    at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:294)

EDIT 2

Running netstat when trying to connect from my local machine to ServerA I get this output:

netstat -a | grep 9010
tcp        0      0 ip-XXX-XX-XX-X.us:34406 ip-XXX-XX-Y-YYY.us:9010 ESTABLISHED
tcp        0      0 localhost:54745         localhost:9010          ESTABLISHED
tcp6       0      0 [::]:9010               [::]:*                  LISTEN     
tcp6       0      0 localhost:9010          localhost:54745         ESTABLISHED

Where X is the same number in both cases and Y is a different number

Solecism

I think the only thing missing here is that while creating the tunnel , you are forwarding a local port, instead of that you should do a remote port forwarding. From local to machine to NAT, i think you should be doing a remote port forwarding, ssh -R 9010:localhost:9010 nat, try both combo, either form you local machine to NAT or to serverA, do a remote port forwarding it should work. To enable remote port forwarding add "GatewayPorts yes" line to /etc/ssh/sshd_config . – Smite

It didn't work, I tried all possible combinations between -R and -L and none of those work. Doing -R to nat and then -R to serverA I get Warning: remote port forwarding failed for listen port 9010 when doing -R to serverA. Doing -R to nat and -L to serverA I get bind: Address already in use channel_setup_fwd_listener: cannot listen to port: 9010. Doing -L to nat and -R to serverA I get connection refused on serverA and doing -L to nat and -L to serverA I get failed connection on jconsole – Solecism

Are you sure you're using the correct port number exposed by NAT rather than the port number of the Glassfish server when connecting to NAT? Look at the NAT routeing table for the index of 9010. – Syllogize

@AakashVerm both the NAT and ServerA are EC2 instances in AWS. I did open all traffic on all ports (both on NAT and ServerA) to my IP on the security groups. Still it didn't work. – Solecism