Does Java 6 open a default port for JMX remote connections?

Question

My specific question has to do with JMX as used in JDK 1.6: if I am running a Java process using JRE 1.6 with

com.sun.management.jmxremote

in the command line, does Java pick a default port for remote JMX connections?

Backstory: I am currently trying to develop a procedure to give to a customer that will enable them to connect to one of our processes via JMX from a remote machine. The goal is to facillitate their remote debugging of a situation occurring on a real-time display console. Because of their service level agreement, they are strongly motivated to capture as much data as possible and, if the situation looks too complicated to fix quickly, to restart the display console and allow it to reconnect to the server-side.

I am aware the I could run jconsole on JDK 1.6 processes and jvisualvm on post-JDK 1.6.7 processes given physical access to the console. However, because of the operational requirements and people problems involved, we are strongly motivated to grab the data that we need remotely and get them up and running again.

EDIT: I am aware of the command line port property

com.sun.management.jmxremote.port=portNum

The question that I am trying to answer is, if you do not set that property at the command line, does Java pick another port for remote monitoring? If so, how could you determine what it might be?

Yokoyokohama

Answer 1

The documentation suggests that the JMX agent uses a local port -- something unreachable from outside the machine -- unless you specify the following property:

com.sun.management.jmxremote.port=portNum

This is for security reasons, as well as for the reason given by Mr Potato Head. Thus, it looks like Java 6 does not open a default remotely accessible port for JMX.

EDIT: Added after the OP added an answer with more information.

Another option you have is to somehow create a local proxy that listens to all local JMX connections and exports this information. This way, you don't need to have such magic configuration of each JVM instance on the server. Instead the local proxy can connect to all JVMs via JMX and then somehow expose this information remotely. I am not positive exactly how you would implement this, but something like this may be less work than what you otherwise have to do to expose all of your JVMs remotely via JMX.

Malayan

Answer 2

AFAIK,

Here are the possibilites for connecting a JMX client process (a management application like jconsole, jmxterm, mc4j, jvmstat, jmxmonitor, jps, ...) to a JMX server process (the agent).

The protocol connecting JMX client and JMX server is assumed to be 'Java RMI' (aka 'RMI-JRMP'). This should be the default. One can configure other protocols, in particular 'RMI-IIOP' and 'JMXMP'. Special protocols are possible: the MX4J project for example additionally provides SOAP/HTTP and various serialization protocols over HTTP.

Refer to Sun/Oracle docs for details on configuration.

Also have a look at the file jre/lib/management/management.properties in your JDK distribution.

So, the possibilities:

Case 0: The JVM is started without any particular configuration

Before Java 6: The JVM does not behave as a JMX server. Any program that is run inside the JVM may access the JVM's MBeanServer programmatically and use it to do interesting data exchanges between threads or to do JVM monitoring, but no management from outside the JVM process is possible.

Since Java 6: Even if not explicitely configured, one can access JMX functionality of the JVM locally (from the same machine) as described in "Case 1".

Case 1: The JVM is started with -Dcom.sun.management.jmxremote

The JVM is configured to work as a local (same-machine-only) JMX server.

In this case (and in principle only for Sun/Oracle JVMs) a JMX client can connect to the JMX server through memory-mapped files found in /tmp/hsperfdata_[user]. This is alluded to in the Sun documentation and called "local monitoring" (and also the Attach API). It does not work on FAT filesystems as permissions cannot be set correctly there. See this blog entry.

Sun recommends running jconsole on a machine separate from the JMX server as jconsole apparently is a resource hog, so this "local monitoring" thing is not necessarily a good idea.

Local monitoring is, however, rather secure, only being usable locally and being easily controlled through filesystem permissions.

Case 2: The JMX server is started with -Dcom.sun.management.jmxremote.port=[rmiregistryport]

The JVM is configured to work as a JMX server listening on several TCP ports.

The port specified on the command line will be allocated by the JVM and an RMI registry will be available there. The registry advertises a connector named 'jmxrmi'. It points to a second, randomly allocated TCP port (an 'ephemeral' port) on which the JMX RMI server listens and through which actual data exchange takes place.

Local as described in 'Case 1' is always enabled in 'Case 2'.

The JMX server listens on all interfaces by default, so you can connect to it (and control it) by locally connecting to 127.0.0.1:[rmiregistryport] as well by remotely connecting to [any outside IP address]:[some port] remotely.

This implies that you have to look at the security implications. You can make the JVM listen on 127.0.0.1:[rmiregistryport] only by setting -Dcom.sun.management.jmxremote.local.only=true.

It is rather unfortunate that one cannot specify where the ephemeral port will be allocated - it is always chosen randomly at startup. This may well mean that your firewall needs to become the swiss cheese of the damned! However, there are workarounds. In particular, Apache Tomcat sets the ephemeral JMX RMI server port via its JMX Remote Lifecycle Listener. The code to perform this little magic can be found at org.apache.catalina.mbeans.JmxRemoteLifecycleListener.

If you use this method, you might as well make sure that:

1. The JMX client has to authenticate to the JMX server
2. The TCP exchange between the client and server is encrypted using SSL

How that is done is described in the Sun/Oracle documentation

Other approaches

You can do interesting permutations to avoid having to use the RMI protocol. In particular, you could add a servlet engine (like Jetty) to your process. Then add servlets that translate some HTTP-based exchange internally into direct accesses to the JVM's MBeanServer. You would then be in 'case 0' but still have management capabilities, possibly through an HTML-based interface. The JBoss JMX Console is an example of this.

More off-topic, you could use SNMP directly (something I haven't tried) according to this document.

Show and Tell Time

And now it's time for some code to illustrate a JXM exchange. We take inspiration from a Sunoracle tutorial.

This runs on Unix. We use a JVM that is configured as a JMX server using:

-Dcom.sun.management.jmxremote.port=9001

We use lsof to check what TCP ports it is holding open:

lsof -p <processid> -n | grep TCP

One should see something like this, the registry port and the ephemeral port:

java    1068 user  127u  IPv6 125614246                 TCP *:36828 (LISTEN)
java    1068 user  130u  IPv6 125614248                 TCP *:9001  (LISTEN)

We use tcpdump to inspect the packet exchange between JMX client and JMX server:

tcpdump -l -XX port 36828 or port 9001

We set up a file .java.policy in the home directory to allow the client to actually connect remotely:

grant {
    permission java.net.SocketPermission 
    "<JMX server IP address>:1024-65535", "connect,resolve";
};

And then we can run this and see what happens:

package rmi;

import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;

import javax.management.remote.rmi.RMIConnection;
import javax.management.remote.rmi.RMIServer;

public class Rmi {

    public static void main(String args[]) throws Exception {
        // We need a Security Manager (not necessarily an RMISecurityManager)
        if (System.getSecurityManager() == null) {
            System.setSecurityManager(new SecurityManager());
        }
        //
        // Define a registry (this is just about building a local data structure)
        // 
        final int comSunManagementJmxRemotePort = 9001;
        Registry registry = LocateRegistry.getRegistry("<JMX server IP address>", comSunManagementJmxRemotePort);
        //
        // List registry entries. The client connects (using TCP) to the server on the
        // 'com.sun.management.jmxremote.port' and queries data to fill the local registry structure.
        // Among others, a definition for 'jmxrmi' is obtained.
        //
        System.out.print("Press enter to list registry entries");
        System.in.read();
        String[] names = registry.list();
        for (String name : names) {
            System.out.println("In the registry: " + name);
        }
        //
        // 'Looking up' the entry registered under 'jmxrmi' involves opening and tearing down
        // a TCP connection to the 'com.sun.management.jmxremote.port', as well as a TCP
        // connection to an ephemeral secondary port chosen at server startup.
        // The actual object locally obtained is a "javax.management.remote.rmi.RMIServerImpl_Stub"
        // indicating where the ephemeral port is.
        // "RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[$IP:$EPHEMERAL_PORT](remote),objID:[-62fb4c1c:131a8c709f4:-7fff, -3335792051140327600]]]]"        
        //
        System.out.print("Press enter to get the 'jmxrmi' stub");
        System.in.read();
        RMIServer jmxrmiServer = (RMIServer)registry.lookup("jmxrmi");
        System.out.println(jmxrmiServer.toString());
        //
        // Now get a "RMI Connection" to the remote. This involves setting up and tearing
        // down a TCP connection to the ephemeral port. 
        //        
        System.out.print("Press enter to get the 'RMIConnection'");
        System.in.read();
        RMIConnection rcon = jmxrmiServer.newClient(null);
        //
        // Ask away. This involves setting up and tearing
        // down a TCP connection to the ephemeral port. 
        //
        System.out.print("Press enter to get the 'domains'");
        System.in.read();
        for (String domain : rcon.getDomains(null)) {
            System.out.println("Domain: " + domain);
        }
        //
        // Ok, that will do. For serious applications, we better use the higher-level JMX classes
        //
    }   
}

Preconcert

I also had to add -Djava.rmi.server.hostname=127.0.0.1 otherwise the default hostname was 127.0.1.1. – Ladner

Sun recommends running jconsole on a machine separate from the JMX server as jconsole apparently is a resource hog... This is very interesting. Can you please provide a source for this information? – Elfreda

Answer 3

Actually there is an undocumented property that you can use to force JMX create remotely accessible connectors on random port numbers.

-Dcom.sun.management.jmxremote.authenticate="false" 
-Dcom.sun.management.jmxremote="true" 
-Dcom.sun.management.jmxremote.ssl="false" 
-Dcom.sun.management.jmxremote.port="0"
-Dcom.sun.management.jmxremote.local.only="false"

Last two properties are of most importance.

Neona

Answer 4

The documentation seems to indicate that the JMX agent uses a local ephemeral port, unless you specify the following property:

com.sun.management.jmxremote.port=portNum

Default ports are avoided because you could have many java applications on one system, and if there was a default port, only one application would be able to be managed! The above configuration property is provided for the express purpose of remote management.

If you must insist on using an ephemeral port, then the URL of the JMX agent should be accessible from within the JVM, through the following system property (although this is likely to be a local address):

com.sun.management.jmxremote.localConnectorAddress

Note: I guess you could always open a socket on a remotely-available address and proxy requests on to the local socket, but using the available option seems far more attractive!

Adrianadriana

Are you saying that an anonymous port (aka effectively a random number) is opened? If so, do you know the range in which we can expect the port to exist? – Yokoyokohama

I would expect it to be an unprivileged (e.g. > 1024) port. – Adrianadriana

It can't be an unprivileged port, as that is reachable from outside if you can guess the port number. – Malayan

@Eddie: The port number has no bearing on whether or not it is reachable externally, only the network interface. When I used the term [un]privileged, I meant it as per e-articles.info/e/a/title/Privileged-Ports-of-a-UNIX-machine. – Adrianadriana

@Mr Potato Head: I understood what you meant by unprivileged. I also understand that the port number has no bearing on whether it is externally reachable. However, you're right that you can open a port (privileged or not) in such a way that it is not externally reachable, only locally. – Malayan

Answer 5

So, the short answer to my question is "no."

However, it's interesting to examine why. Look at the netstat output from a valid local connection. Here are the ports that I see opened up as a result of a jconsole making a local connection to itself. As you can see, port 1650 is the local port being used for the JMX information:

Proto  Local Address          Foreign Address        State
TCP    Gandalf:1650           Gandalf:1652           ESTABLISHED
TCP    Gandalf:1650           Gandalf:1653           ESTABLISHED
TCP    Gandalf:1650           Gandalf:1654           ESTABLISHED
TCP    Gandalf:1650           Gandalf:1655           ESTABLISHED
TCP    Gandalf:1650           Gandalf:1656           ESTABLISHED
TCP    Gandalf:1652           Gandalf:1650           ESTABLISHED
TCP    Gandalf:1653           Gandalf:1650           ESTABLISHED
TCP    Gandalf:1654           Gandalf:1650           ESTABLISHED
TCP    Gandalf:1655           Gandalf:1650           ESTABLISHED
TCP    Gandalf:1656           Gandalf:1650           ESTABLISHED

However, it's not sufficient to try to connect jconsole to localhost:1650. Sadly, all that will net you is a "Connection failed: no such object in table" message.

So, the conclusion of my original story is that, if we are going to facilitate remote monitoring using JMX for our customers, we really need to identify unique individual remote access ports for the variety of Java processes that are started in our system. Fortunately, all this requires is the judicious use of the VM argument:

com.sun.management.jmxremote.port=portNum

where we will almost certainly have a sequential pre-specified range of portNum so that the customer can select the correct remote application using the port number.

Yokoyokohama

Yes, this is what I said in my answer, yes? – Malayan

Which is also probably a good idea, rather than relying on the default staying the same. – Nucleolus

Answer 6

I've been working recently to figure out how to enable remote JMX management from java code, without requiring the JVM to have been started with special properties set. The solution I settled on is to start my own private RMI registry -- easy enough -- and to expose the JMX service on that registry. I create my own MBeanServer, then create a new JMXConnectorServer. The JMXConnectorServer is created through a call like

connector = JXMConnectorServerFactory.newJMXConnectorServer(url, null, server);

Where server is the MBeanServer, and url is an instance of JMXServiceURL.

The url is of the form "service:jmx:rmi:///jndi/rmi://localhost:/jmxrmi" where port is the (local) private registry's port number. "jmxrmi" is the standard service name for the JMX service.

After setting this up, and starting the connector, I find that I can connect to it from jconsole using hostname:port.

This entirely addresses my needs; I will be interested to know if anyone sees a flaw in this approach.

Reference: JMX Tutorial, Chap. 3

Cosma

Answer 7

If you happen to run your application inside Glassfish app server, simply run the following asadmin command, you would need to restart all running servers for the change to take affect.

./asadmin enable-secure-admin

There are extra Glassfish server configurations to further enable security, see more at Connecting remotely to Glassfish through JMX.

Lacunar

You do need to follow the steps on JVM options, these are instructions for Java applications in general. For a Java application that runs with Glassfish application server, simply using the asadm command. That saved me lots of time! – Lacunar

Another note, with only the settings using JVM options, you would still see run time exception, and Glassfish would fail to start. – Lacunar