How to access Spring-boot JMX remotely

Question

I know that spring automatically expose JMX beans. I was able to access it locally using VisualVM.

However on prod how I can connect to remotely to the app using it's JMX beans? Is there a default port or should I define anything in addition?

Thanks, ray.

Store

Answer 1

By default JMX is automatically accessible locally, so running jconsole locally would detect all your local java apps without port exposure.

To access an app via JMX remotely you have to specify an RMI Registry port. The thing to know is that when connecting, JMX initializes on that port and then establishes a data connection back on a random high port, which is a huge problem if you have a firewall in the middle. ("Hey sysadmins, just open up everything, mkay?").

To force JMX to connect back on the same port as you've established, you have a couple of the following options. Note: you can use different ports for JMX and RMI or you can use the same port.

Option 1: Command line

-Dcom.sun.management.jmxremote.port=$JMX_REGISTRY_PORT 
-Dcom.sun.management.jmxremote.rmi.port=$RMI_SERVER_PORT

If you're using Spring Boot you can put this in your (appname).conf file that lives alongside your (appname).jar deployment.

Option 2: Tomcat/Tomee configuration

Configure a JmxRemoteLifecycleListener:

Maven Jar:

    <dependency>
        <groupId>org.apache.tomcat</groupId>
        <artifactId>tomcat-catalina-jmx-remote</artifactId>
        <version>8.5.9</version>
        <type>jar</type>
    </dependency>

Configure your server.xml:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
      rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" />

Option 3: configure programmatically

@Configuration
public class ConfigureRMI {

    @Value("${jmx.rmi.host:localhost}")
    private String rmiHost;

    @Value("${jmx.rmi.port:1099}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi://%s:%s/jndi/rmi://%s:%s/jmxrmi", rmiHost, rmiPort, rmiHost, rmiPort));
        return connectorServerFactoryBean;
    }
}

The trick, you'll see, is the serviceUrl in which you specify both the jmx:rmi host/port and the jndi:rmi host/port. If you specify both, you won't get the random high "problem".

Edit: For JMX remoting to work, you'll need to make a decision about authenticating. It's better to do it in 3 distinct steps:

1. basic setup with -Dcom.sun.management.jmxremote.authenticate=false then
2. add a password file (-Dcom.sun.management.jmxremote.password.file). See here for instructions. + -Dcom.sun.management.jmxremote.ssl=false and then
3. set up SSL.

Utilitarian

Does this approach exclude using JAVA_OPTS method below? – Discrimination

I tried this method (Option 3) with a Java service running within a Docker container, and had trouble connecting to it from outside the container. I'll have to do a small sample project to explore this idea better. – Naughton

is this (appname).conf file documented somewhere? – Robin

@OleksandrSh, the documentation is here: docs.spring.io/spring-boot/docs/2.3.2.RELEASE/reference/html/… – Intercellular

Option 1 errors out (JDK 11, Spring Boot 2.3.2) with Error: Password file not found: {skipped} jdk.internal.agent.AgentConfigurationError at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:572) at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:436) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) – Intercellular

Well, Option 1 (-Dcom.sun.management.jmxremote.port=$JMX_REGISTRY_PORT + -Dcom.sun.management.jmxremote.rmi.port=$RMI_SERVER_PORT) is missing the required property: -Dcom.sun.management.jmxremote.authenticate=false or true with the setup from @Popyeye answer. It's just not working without this in OpenJDK 11. That's the root cause for the Error: Password file not found. – Intercellular

@DKroot Java 11 is no different than Java 7+ in this regard. The authenticate flag is true by default and you're supposed to configure a jmxremote.password file. Setting the flag to false is a security vulnerability so don't do it unless you're on a non-prod machine. – Utilitarian

Yes, I went with @Popeye's setup. Overall, JMX configuration was incredibly hairy. I think it's better to do it in 3 distinct steps: 1) basic setup with -Dcom.sun.management.jmxremote.authenticate=false then 2) add a password file and -Dcom.sun.management.jmxremote.ssl=false and then 3) set up SSL – Intercellular

@DKroot That's reasonable advice. Feel free to update the ticket to say so if you like, or I can if you'd prefer not to. – Utilitarian

@Utilitarian Sure. Which ticket would I need to update? – Intercellular

@DKroot Ha! I said "ticket" out of habit. I meant add your advice to the SO Answer above. – Utilitarian

@Utilitarian Done. OMG: you can edit somebody else's answer! It's the first time I did it on SO. My own answers: sure. Somebody else's: blew my mind. :) – Intercellular

Answer 2

Add the following JVM Properties in "$JAVA_OPTS" (in your application):

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<PORT_NUMBER> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<HOST'S_IP>

In the Jconsole/Visual VM use the following to connect:

service:jmx:rmi:///jndi/rmi://<HOST'S_IP>:<PORT_NUMBER>/jmxrmi

It doesn't enable security, but will help you to connect to the remote server.

Marjorie

This did the trick for me ... thank you. It did not work until I added the -Djava.rmi.server.hostname. I was able to connect via jconsole like this: ip:port – Ellga

These params have to be used via the java command line. Setting them in JAVA_OPTS didn't work for me for Spring Boot 2.3.2. Here is a longer blog post: medium.com/@cl4r1ty/…. (Docker does not matter, JAVA_OPTS="..." java -jar doesn't work either.) – Intercellular

Answer 3

A tested approach on Java 1.8.0_71 and Spring Boot(1.3.3.RELEASE). Append below parameters to JVM arguments for monitored JVM.

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12348 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=12349 -Dcom.sun.management.jmxremote.password.file=/somewhere/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/somewhere/jmx/jmxremote.access

The com.sun.management.jmxremote.port is used to define the fixed RMI registry port, and the com.sun.management.jmxremote.rmi.port is used to instruct JVM to use fixed RMI port, but NOT use random one.

By setting this, I am able to connect JVM client from remote host to the monitored JVM via a firewall just opening 12348 and 12349 port.

I tested using java -jar cmdline-jmxclient-0.10.3.jar user:pwd hostip:12348 on a remote machine, which generates below output(shortened just for demonstration).

java.lang:type=Runtime
java.lang:name=PS Scavenge,type=GarbageCollector
Tomcat:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,j2eeType=Filter,name=requestContextFilter
java.nio:name=mapped,type=BufferPool
Tomcat:host=localhost,type=Host
java.lang:name=Compressed Class Space,type=MemoryPool
.......

The jar is downloaded from Here.

Arta

I tested this approach successfully in OpenJDK 11 with 2 improvements: 1. Using the same port for JMX and RMI. 2. Omitting -Dcom.sun.management.jmxremote: this is no longer needed. – Intercellular

Answer 4

Another alternative

Reference for jmxremote.password and jmxremote.access files

import java.util.HashMap;
import java.util.Map;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.jmx.support.ConnectorServerFactoryBean;
import org.springframework.remoting.rmi.RmiRegistryFactoryBean;

@Configuration
public class ConfigureRMI {

    @Value("${jmx.rmi.password.file:/tmp/jmxremote.password}")
    private String passwordFile;

    @Value("${jmx.rmi.access.file:/tmp/jmxremote.access}")
    private String accessFile;

    @Value("${jmx.rmi.port:19999}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        Map<String, Object> properties = new HashMap<>();
        properties.put("jmx.remote.x.password.file", passwordFile);
        properties.put("jmx.remote.x.access.file", accessFile);
        connectorServerFactoryBean.setEnvironmentMap(properties);
        connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi:///jndi/rmi://:%s/jmxrmi", rmiPort));
        return connectorServerFactoryBean;
    }
}

Allegedly