NoRouteToHostException / NoSuchHostException on remote JMX call

Question

I'm having trouble making a remote JMX call to JBoss 6 on a Centos 5.6 server. I've previously been able to do this when running the same app on a Debian server.

./twiddle.sh --server=service:jmx:rmi:///jndi/rmi://SERVER:1090/jmxconnector invoke foo:service=bar baz

Caused by: java.net.NoRouteToHostException: No route to host
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.<init>(Socket.java:375)
    at java.net.Socket.<init>(Socket.java:189)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)

The same call succeeds if I run it locally on SERVER. I've opened port 1090 with iptables, and I can connect via telnet to SERVER:1090. hostname -i returns the correct IP address.

I've also tried starting JBoss with -Djava.rmi.server.hostname=localhost. If I do this, then I get a different exception:

Caused by: java.rmi.NoSuchObjectException: no such object in table
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
    at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
    at org.jboss.console.twiddle.Twiddle.createMBeanServerConnection(Twiddle.java:322)
    at org.jboss.console.twiddle.Twiddle.connect(Twiddle.java:331)
    at org.jboss.console.twiddle.Twiddle.access$400(Twiddle.java:60)
    at org.jboss.console.twiddle.Twiddle$1.getServer(Twiddle.java:217)

Benelux

What happens if you make the JMX call using the IP address of the server? – Lyonnaise

@SamGoldberg The same thing, unfortunately – Benelux

Answer 1

It turns out that in addition to port 1090, JMX/RMI also uses a dynamically allocated port, which gets blocked by the firewall. So, if it's appropriate, disable the firewall altogether, or else this seems to be an alternative (which I've not tried yet):

http://olegz.wordpress.com/2009/03/23/jmx-connectivity-through-the-firewall/

Benelux

An alternative is to setup a kind of VPN to route the full port range to a specific IP address through a tunnel. A customer of mine have implemented their own JVM agent in C to be able to tunnel JMX protocol through SSH... So really a common trouble. – Ramiroramjet

Answer 2

What I did was this:

Update the file activemq.xml and specify rmiServerPort.

<managementContext createConnector="true" connectorPort="SOME_PORT" rmiServerPort="SOME_OTHER" jmxDomainName="org.apache.activemq"/>

Allow both ports by updating your iptables entry, restart activemq and it should work.

Prehensible