Allow restarting Java application with JMX monitoring enabled immediately

Question

I have a Java application with JMX monitoring enabled like this:

-Dcom.sun.management.jmxremote.port=9999 \
// some other properties omitted

But when I try to restart the application, sometime I got an error says the JMX port number is already in use. This is not acceptable.

So I want to set the SO_REUSEADDR to true for the underlying socket to avoid this error but found no related JMX properties.

Any idea?

Barrelchested

Have you looked to see what application is using that port? – Tineid

I must be my application. When I stop the application, I think the socket bound to this port goes to TIME_WAIT state for 2MSL to close actually. So I want to make this port reusable. – Barrelchested

SO_REUSEADDR does not work that way. It allows sockets to listen to specific IP addresses and ignore others. Either the same application is running twice or there is another application grabbing this port. – Tineid

@BevynQ, if I stop the application and do NOT start it immediately and later I try to start the application, I works. So I think it's the socket TIME_WAIT mechanism disallow – Barrelchested

are you open to different approaches ? – Erne

@Barrelchested You say you "think" it is. Is it or isn't it? What process does netstat say is using that port, and what state is it in? – Cruller

@Tineid i doublechecked and I believe OP is correct in what SO_REUSEADDR does. – Tetzel

@NeilCoffey netstat does not name a process for "TIME_WAIT" if the process is already finished, e.g. tcp 0 0 localhost:57525 localhost:3100 TIME_WAIT -. There are also some questions about SO_REUSEADDR in general (like https://mcmap.net/q/103298/-what-is-the-meaning-of-so_reuseaddr-setsockopt-option-linux-duplicate/602119) which also mention TIME_WAIT in the answers. – Fontes

Have you tried ServerSocket.setReuseAddress(true)? – Deprecatory

The jmx server-port is created by the JVM, but a fairly easy way to create the server-socket myself would be a good workaround or even a solution. – Fontes

@Fontes Ah sorry OK, if it doesn't, my bad -- I thought it did still list it even if there was no process attached (maybe this varies?) – Cruller

Answer 1

I am afraid you can't do that from command line.

You will need to create a RMIServerSocketFactory, which produces ServerSockets with the desired option (SO_REUSEADDR).

Docs here: http://docs.oracle.com/javase/8/docs/technotes/guides/rmi/socketfactory/

Someone else solving the same problem: https://svn.apache.org/viewvc?view=revision&revision=r1596579

Tetzel

Thanks. I'm afraid that might indeed be the only solution. Putting it all together it's much more verbose than I'd hoped, but still better then nothing. – Fontes

Answer 2

I had the same issue. It was the first instance of my application (which I had stopped) which was still subscribed to this port, so the new instance could not start. In my case it didn't have to do with the socket TIME_WAIT mechanism but rather with the fact that after calling stop(), it took some time until all running threads ended gracefully. What worked in my case was unregistering the bean right before stopping the application so that the socket is free.

private void unregisterBeanForName(String name) {
        try {

            JMXServiceURL jmxServiceURL = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");
            JMXConnector cc = JMXConnectorFactory.connect(jmxServiceURL);
            MBeanServerConnection mbsc = cc.getMBeanServerConnection();
//This information is available in jconsole
            ObjectName serviceConfigName = new ObjectName("YourObjectName");
            mbsc.unregisterMBean(serviceConfigName);
// Close JMX connector
            cc.close();
        } catch (Exception e) {
            System.out.println("Exception occurred: " + e.toString());
            e.printStackTrace();
        }
    }

Twana

Answer 3

Yes, you should create JMX connector programmatically. As easier workaround, you can select another port for JMX in runtime if default port is busy by just killed process. OR just attempt opening your port again and again until it succeeds.

Here is code snippet i use to open JConsole-compatible JMX connector. In Scala, sorry, but you should be able to adapt it easily enough

def startJmx(port: Int): Unit = {
if (port < 1) {
  return
}

val log = LoggerFactory.getLogger(getClass)

log.info("Starting JMX server connector on port {}", port)

val registry = LocateRegistry.createRegistry(port)

val server = ManagementFactory.getPlatformMBeanServer()

val url = new JMXServiceURL(s"service:jmx:rmi:///jndi/rmi://localhost:$port/jmxrmi")

val connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(url, Collections.emptyMap(), server)

 val thread = new Thread {
   override def run = try {
     connectorServer.start()
   } catch { 
     case e: Exception => log.error("Unable to start JMX connector", e)
   }
 }
 thread.setDaemon(true)
 thread.setName("JMX connector Thread")
 thread.start()
}

Siusiubhan

Answer 4

This could be one workaround: On the remote server you can have two ports: 9999 and 9998 forwarded to 9999.

When restarting your application alternate a boolean each time to decide to connect to 9999 or 9998.

Jitney

Answer 5

Add a shutdown hook on your application that will kill jmx.

// kill process with port 9999    
fuser -k 9999/tcp

Galenical

This would be platform-dependent (which would be OK in my case) but more importantly it doesn't work. Killing the process via fuser does not seem to change the behaviour regarding SO_REUSEADDR. I tried with a simple class and still got the BindException – Fontes