Improving JBoss 5 slow startup

Asked 18/12, 2009 at 8:46 Answered 16/11, 2012 at 19:40

We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.

18:53:35,444 INFO  [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms

It seems like the component that is taking JBoss the longest time to initialize is the JMX

18:50:41,926 INFO  [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl@1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO  [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector

From the DEBUG server log, I get these lines at the problematic time:

2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData@1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false

There is no EJB in the project.

The memory settings are:

 -Xms128m -Xmx512m -XX:MaxPermSize=256m

Do you have any idea how JBoss start time can be improved?

Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant

Harewood answered 18/12, 2009 at 8:46 Comment(4)

Are you certain it has enough memory and doesn't start swapping when JMX is initialized? – Glomerulonephritis 18/12, 2009 at 9:45

Do you use EJB in your project? – Bevins 18/12, 2009 at 12:16

Did mhaller's answer actually help? By how much? – Sellingplater 25/3, 2010 at 12:10

I'm also interested in you did after reading mhaller's answer. My JBoss start up is slowest at the same spot. Not able to upgrade to the latest JBoss yet. – Antiperiodic 22/4, 2011 at 8:21

A shot into the blue sky without more information

Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).

Maryalice answered 27/12, 2009 at 0:0 Comment(2)

I was having issues with the "wrong localhost" lookup and my integration test would pause for up to a minute. Setting -Djava.rmi.server.hostname=127.0.0.1 solved it for me, thanks! – Announcer 20/11, 2014 at 21:11

I had a similar issue to the hostname, where my /etc/hosts file had the name of my dev machine listed twice. This was enough to slow down the load from about 30 seconds to about 5 minutes. – Homolographic 24/2, 2016 at 16:23

That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.

The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss

Ardenia answered 18/12, 2009 at 8:49 Comment(2)

well, it's not consistent, I changed the log level to DEBUG and now it takes 1.5 minutes to start :-). once the problem is reproduced I will post the debug logs – Harewood 18/12, 2009 at 9:5

Hmm, still not much useful info in the log. Try TRACE logging? :) – Ardenia 20/12, 2009 at 9:43

Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.

While I'm no expert, I would suggest the following:

Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.

Liebknecht answered 27/1, 2010 at 1:58 Comment(0)

I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:

 -Xms512m -Xmx512m

With this, I improved from 4 to 2 minutes the starting time.

Ducan answered 17/11, 2010 at 23:9 Comment(1)

for a better explanation check this link: petefreitag.com/articles/gctuning at section: 3.1 Total Heap – Ducan 17/11, 2010 at 23:13

You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start

Mopes answered 24/12, 2009 at 12:34 Comment(0)

See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.

Isolde answered 28/2, 2011 at 12:28 Comment(0)

I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s

15:00:26,449 INFO  [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms

I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)

Pelotas answered 21/12, 2009 at 18:3 Comment(1)

please put an explanation with your down vote (just to learn from it) – Pelotas 6/1, 2010 at 14:9

I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode. But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:

I've solved that. It's a debugging issue. I've changed my debug settings from:

wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n

to:

wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n

and JBoss becomes fast again.

Hope this helps.

Partridgeberry answered 23/12, 2009 at 16:34 Comment(4)

this one looks promising, indeed we switched from JDK 5 to JDK6. But what do you mean change settings, where do I find these debug settings? – Harewood 23/12, 2009 at 18:34

The main point are the options you pass to the JVM when starting JBOSS. So you shouldn't start the JVM with options -Xnoagent and -Djava.compiler=NONE when using -Xdebug. (see the missing lines in second part) The above sample is about the Java Service Wrapper (wrapper.tanukisoftware.org/doc/english/…). Here is an explanation about running JBOSS as a service using JSW: community.jboss.org/wiki/RunJBossAsAServiceOnWIndows. – Partridgeberry 23/12, 2009 at 19:38

WEHERE IS THE PROPERTIES FILE TO SET THIS INFORMATION!?!?!? – Horehound 23/6, 2010 at 10:4

That would be either in the JBoss bin directory (run.bat, run.conf.bat), or in your IDE if you're running it on your dev machine. – Nisa 21/7, 2011 at 16:29

Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.

Discolor answered 26/12, 2009 at 23:43 Comment(0)

Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.

I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.

Waggoner answered 16/11, 2012 at 19:40 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags