Why Java opens 3 ports when JMX is configured?

Question

I run my Java program with JDK7 on Centos6. I enable JMX using the following options:

JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9123 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true"

When I check what ports are opened I discover 2 additional random ports:

netstat -plunt | grep java
tcp        0      0 :::9123                     :::*                        LISTEN      13295/java
tcp        0      0 :::59927                    :::*                        LISTEN      13295/java
tcp        0      0 :::59928                    :::*                        LISTEN      13295/java

Please note that each restart only configured port 9123 remains same, and two additional ports change values.

netstat -plunt | grep java
tcp        0      0 :::9123                     :::*                        LISTEN      13331/java
tcp        0      0 :::59932                    :::*                        LISTEN      13331/java
tcp        0      0 :::59933                    :::*                        LISTEN      13331/java

What are 2 additional ports and why they are opened?

How can I configure 2 additional random ports?

How can I configure ::ffff:127.0.0.1 will appear before all ports opened by JMX?

Why one port is not used when connecting with JConsole?

Added to clarify the answer

Unfortunately, the additional random port is still opened To remind you, I use Centos 6. My Tomcat settings are look like this (Tomcat does not deploy any applications):

CATALINA_OPTS="${CATALINA_OPTS}  -XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.useLocalHostname=true -Djava.rmi.server.useCodebaseOnly=true -Dcom.sun.management.jmxremote.port=9123 -Dcom.sun.management.jmxremote.rmi.port=9123"

Tomcat process looks like this:

/usr/java/jdk1.7.0_51/bin/java -Djava.util.logging.config.file=/usr/tomcat-7.0.47/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.useLocalHostname=true -Djava.rmi.server.useCodebaseOnly=true -Dcom.sun.management.jmxremote.port=9123 -Dcom.sun.management.jmxremote.rmi.port=9123 -Djava.endorsed.dirs=/usr/tomcat-7.0.47/endorsed -classpath /usr/tomcat-7.0.47/bin/bootstrap.jar:/usr/tomcat-7.0.47/bin/tomcat-juli.jar -Dcatalina.base=/usr/tomcat-7.0.47 -Dcatalina.home=/usr/tomcat-7.0.47 -Djava.io.tmpdir=/usr/tomcat-7.0.47/temp org.apache.catalina.startup.Bootstrap start

Unfortunately, each time I see additional listening port:

tcp        0      0 :::38830                    :::*                        LISTEN      790/java
tcp        0      0 ::ffff:127.0.0.1:8080       :::*                        LISTEN      790/java
tcp        0      0 :::9123                     :::*                        LISTEN      790/java

Additional run:

tcp        0      0 ::ffff:127.0.0.1:8080       :::*                        LISTEN      2348/java
tcp        0      0 :::36252                    :::*                        LISTEN      2348/java
tcp        0      0 :::9123                     :::*                        LISTEN      2348/java

BTW, why I can not see ::ffff:127.0.0.1 before RMI ports?

Added second time to clarify the comment

It is not related to Tomcat. I have tried to run ant with similar settings: Ant process looks like this:

/usr/bin/java -XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.useLocalHostname=true -Djava.rmi.server.useCodebaseOnly=true -Dcom.sun.management.jmxremote.port=9123 -Dcom.sun.management.jmxremote.rmi.port=9123 -classpath /usr/apache-ant-1.9.2/lib/ant-launcher.jar -Dant.home=/usr/apache-ant-1.9.2 -Dant.library.dir=/usr/apache-ant-1.9.2/lib org.apache.tools.ant.launch.Launcher -cp  sleep

Unfortunately, each time I see additional listening port:

tcp        0      0 :::41200                    :::*                        LISTEN      13597/java
tcp        0      0 :::9123                     :::*                        LISTEN      13597/java

Additional run:

tcp        0      0 :::58356                    :::*                        LISTEN      13629/java
tcp        0      0 :::9123                     :::*                        LISTEN      13629/java

Answer: It is Java's bug

I success to open bug on Java: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8035404

Lawler

possible duplicate of JMX enabled Java application appears to open a random high order port when JMX client connects – Antakiya

Not. Nobody explains why 3-d port is opened (than not used by JConsole) – Lawler

Try following the links in the answer - this one is particularly useful. The short answer is "that's how JMX works". – Antakiya

I read these answers before the question was created. They explain about one additional port. Nothing regarding 3-d port :( I encounter it when I run Tomcat: #20699568 If you know why 3-d port is opened - please share I will be happy. – Lawler

And what will change when you know? Its still going to open three ports because "that's how JMX works". IMO this is like asking why Excel keeps a lock on the file that you have open in it; it just works that way. Knowing the exact reason (which you'd have to ask the original designer) is not going to change anything in the way it works. – Fillender

If they open the port by mistake I will open a bug on Java :) With Excel it is not the bug. – Lawler

What reason do you have to believe it's a mistake? – Birthwort

I success to open bug on Java: bugs.java.com/bugdatabase/view_bug.do?bug_id=8035404 – Lawler

@Lawler the ticket was resolved as "Not an issue". Have you managed to figure out why there was such behaviour? – Yazzie

Sorry to see this was not considered an issue. – Honestly

Any luck with the 3rd port folks ? We have scanners installed which identify opened ports through all servers. Problem is, the 3rd port that is opened is purely random, so every tomcat restart opens a different 3rd port. This causes false positives, and we can't even whitelist a specific number! – Liaoyang

Answer 1

Contrary to common belief JMX/RMI doesn't need to open all these ports. You can actually force them to be same which will mean that at the end of the day you'll only need to punch one hole in the firewall (if firewall is your concern).

Try setting System Properties:

com.sun.management.jmxremote.port
com.sun.management.jmxremote.rmi.port

to the same value!!

Explicitly setting these will stop RMI from picking random ports. Setting them to the same value will make sure it opens less ports to listen on.

This will work in Java 7 update 25 or later.

What is the third port?

The third port that you see opened by your application (or the second if you followed my advice above) is used by the Java Attach API. It is what JConsole uses for connecting to "Local Process". The Java Attach API feature is enabled by default since Java 6 regardless of the com.sun.management.jmxremote property. This feature will use a random port (aka an OS ephemeral port) but it really doesn't matter because the feature only allows connections from the host itself. If you really dislike this feature then you can add -XX:+DisableAttachMechanism to the command line to disable the Java Attach API feature. Then you'll no longer see the java process (in this case Tomcat) listening on a random port.

How do I make JMX listen on the loopback interface only

With a custom made application you would use a RMIServerSocketFactory but this is Tomcat so you would have to do it using Tomcat's JMX Remote Lifecycle Listener.

On the other hand it doesn't matter now that you have the com.sun.management.jmxremote.local.only property since Java 7. It makes sure that only connections from the host itself are allowed. Mind you that JMX library doesn't achieve this by binding to loopback interface which would certainly be one way of doing it but also slight inaccurate as a host can potentially have several loopback interfaces.

In fact by and large (with the most recent additions to JDK wrt JMX) I would say that Tomcat's JMX Remote Lifecycle Listener is now redundant except if you want to bind to some really odd network interface.

Nakia

Thanks (+1)! I have tried it on 1.7.0_51 without success. The suggested configuration allows to configure one more port, but unfortunately one port remains random. Do you know how to configure the last port (or close it)? – Lawler

So, if I understand you correctly you managed to cut it down from 3 listening ports to 2 listening ports? I've updated my answer with more information. – Nakia

So what remains is that -XX:+DisableAttachMechanism doesn't seem to have effect ? I must admit I only tested it on a standalone app, not on Tomcat. – Nakia

It is not related to Tomcat. I tested it also using Ant (see updated answer). BTW, why I can not see ::ffff:127.0.0.1 before RMI ports? – Lawler

@Michael: why do you expect it only to bind to loopback interface? (I assume you have this expectation because of the com.sun.management.jmxremote.local.only property ?). Read my answer above in particular the section entitled "How do I make JMX listen on loopback interface only". – Nakia

I have opened bug on Java: bugs.java.com/bugdatabase/view_bug.do?bug_id=8035404 – Lawler

Out of curiosity, how did you find out about the com.sun.management.jmxremote.rmi.port property? It's currently undocumented: bugs.openjdk.java.net/browse/JDK-8129948 – Despoil

@DanielSerodio. I read the JDK source. You'll find it in sun.management.jmxremote.ConnectorBootstrap.java. – Nakia

Thanks @peterh. I tried setting both to the same port and got java.rmi.server.ExportException: Port already in use: 1099. Comments on bugs.java.com/bugdatabase/view_bug.do?bug_id=8035404 say the two ports need to be separate. – Despoil

AFAICT com.sun.management.jmxremote.local.only=true will not disable remote connections. I've tested this emperically, and it also matches up with the code: github.com/openjdk/jdk/blob/master/src/jdk.management.agent/…, USE_LOCAL_ONLY is only used in the startLocalConnectorServer method, not in startRemoteConnectorServer, which is the one used for... remote connections. – Angularity

"it really doesn't matter because the feature only allows connections from the host itself" it does matter if the randomly selected port happens to conflict with the port another application on the same host listens on. – Probity

@Thayne. Operating systems have a concept called "ephemeral port". This is what is used here. So yes, it is random, but only within a range, typically 49152 to 65535 depending on the OS. So Java/RMI basically says to the OS: "please give me random listening socket in the ephemeral range" and the OS then does that. There's no risk of collision. The OS keeps track of which ephemeral ports it has already handed out and which has been released. – Nakia

I still can see third port opened and listening on all interfaces, which is very annoying. I checked the bug URL and nothing useful there, as it says "Not an issue". Also, -XX:+DisableAttachMechanism didn't gave me anything. To be clear here, I am dealing with Kafka, and really want to have all its ports under control. Any suggestions on how to disable of bind that Java Attach API to a specific interface. Google didn't gave me anything usefull. – Yazzie

@Max, were you able to resolve the third port issue? – Confab

Why do we need 2 ? com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port? Why not just 1 ? – Candicecandid

@RamPrakash. This is what my answer says: you can set these properties to the same value and then you'll achieve that it is only one port ... and it will work. – Nakia

No sir.. I have been googling to understand the difference between com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port . What is the need for these properties in the first place. unfortunately everyone says pass this value. but could not find a clear source which explains these properties. Interestingly this answer comes top in google results. you can add that info too if you think it could help others. – Candicecandid

Per a comment at the bottom of bugs.java.com/bugdatabase/view_bug.do?bug_id=8035404 you are not allowed to set both com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port to the same value. – Cerargyrite

Answer 2

Using Oracle Java SE 1.8.0_121.

It's possible to set jmxremote.port and jmxremote.rmi.port to the same value, it's one less port opened. It's also possible to set jmxremote.host=127.0.0.1, to have that port (or those two ports, if you set them differently) bind to the loopback interface only.

Another port is still dynamically assigned though, and will bind to 0.0.0.0. I was not able to prevent this port with -XX+DisableAttachMechanism, and was also unable to make it bind to anything else than 0.0.0.0.

Numbat

Although this is not an answer, I can confirm the described behaviour. – Honestly

@user2679436, were you able to resolve the third port issue? – Confab

@Confab I was able to reduce the fixed ports to a single one using the method described above. But there was nothing I could do about the dynamic port on 0.0.0.0, have to live with it. – Numbat

@user2679436, thank you for the update. I too see that it can't be avoided. But a bug filed indicates that an option was provided in JDK 15 to make the port fixed. I haven't tried. bugs.openjdk.java.net/browse/JDK-8237631 – Confab

Answer 3

Because jmx is encapsulated in rmi wich is very firewall and nat unfriendly. Avoid it if you can, there is an alternative encapsulation called jmxmp.

Have a look at, that might help you : http://blog.markfeeney.com/2010/10/jmx-through-ssh-tunnel.html http://jrds.fr/sourcetype/jmx/start#jmx_protocols

Conners

'Firewall and NAT unfriendly' doesn't explain the actual ports. – Birthwort

Answer 4

The third port has nothing to do with how RMI (JRMP) works, as incorrectly noted in the bug.

Also has nothing to do with the Java Attach API.

It is the port used by the local JMX listener. Whenever remote JMX is started, for some reason the local one is started as well. This can be seen in this code from OpenJDK:

        /*
         * If the jmxremote.port property is set then we start the
         * RMIConnectorServer for remote M&M.
         *
         * If the jmxremote or jmxremote.port properties are set then
         * we start a RMIConnectorServer for local M&M. The address
         * of this "local" server is exported as a counter to the jstat
         * instrumentation buffer.
         */
        if (jmxremote != null || jmxremotePort != null) {
            if (jmxremotePort != null) {
                jmxServer = ConnectorBootstrap.
                        startRemoteConnectorServer(jmxremotePort, props);
                startDiscoveryService(props);
            }
            startLocalManagementAgent();
        }

It is also mentioned in the documentation:

To enable monitoring and management from remote systems, you must set the following system property when you start the Java VM: com.sun.management.jmxremote.port=portNum

In the property above, portNum is the port number through which you want to enable JMX RMI connections. Be sure to specify an unused port number. In addition to publishing an RMI connector for local access, setting this property publishes an additional RMI connector in a private read-only registry at the specified port using a well known name, "jmxrmi".

Also, since 2020 the local port can be configured via com.sun.management.jmxremote.local.port.

Meaghanmeagher