Java server cpu usage at 100% after two days continous running with about 110 users

Question

I have a tomcat 6.0.20, apr 1.2, jdk 1.6.0_15 with mysql 5.1.38 running on a rhel box with 4 GB ram. There is one simple jsp/servlet application on it with 5 users, one struts 1.2.0.9 with 64 users on it and one struts 2.0 application with 35 users on it. The struts 2.0 users make an entry every second, about 900 entries in a day. I am also using toplink for persistence in the last two applications. I have declared all non referenced objects to null, in the code, have applied production values for config files from the struts 2 site and from the tomcat site. Applied thread caching in mysql, reduce wait_timeout and interactive_timeout to be equivalent to session timeout of tomcat.Increased file descriptors in linux. Reworked queries. Examined the thread dump, watched gc stats, applied above changes on basis of this,

YET still facing "java.lang.OutOfMemoryError" error.

at different times its for different things, sometimes its Servlet.service(), sometimes its image.servlet, sometimes it jasper that causes it.

extremely frustrtating, as the errors are not constant but keep changing over time

Any help please would be greatly appreciated!!!

JAVA_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+CMSParallelRemarkEnabled (tomcat manager reports 34 mb empty so have not used permsize, mx and mn etc.)

persistence.xml

 <property name="toplink.jdbc.url" value="jdbc:mysql://localhost:3306/dbname?autoReconnect=false"/>

server.xml

<Connector port="80" protocol="HTTP/1.1" connectionTimeout="2000" redirectPort="8443" compression="on" compressableMimeType="application/octet-stream,text/html,text/xml,text/plain,application/x-javascript,image/gif,text/css,image/gif,application/vnd.ms-excel,application/pdf" enableLookups="false"/>

context.xml

<Context reloadable="false" delegate="false" privileged="false">

Ozalid

Are you changing max heap size from the default? I don't see a -Xmx argument. – Delrio

Is it a PermGen problem? PermGen has a limit different from the max heap setting. So it is entirely possible to run out of PermGen space and still have 34MiB unused. – Postilion

And from my experience you need to completely restart JBoss/Tomcat about every third application reload or face a PermGen problem. So now we restart JBoss every time we deploy just to be sure. – Postilion

Answer 1

First, you must make sure which process is eating all the CPU. Is it really the java program? Try top(1) to figure this out if you haven't already.

If you have and you're sure that it's the Java program, enable remote debugging. When the CPU boils over next time, connect and make sure that no thread is in an infinite loop.

If that's not the case, you're out of memory (no matter what the tomcat manager says). Start a JMX console and check the various memory spaces. My guess is that the permgen is pretty full (are there many big JARs on your classpath? Or are you using cglib somewhere?) When that happens (and since you have perm gen GC enabled), the Java VM will try to free memory in the perm gen space all the time.

Unless you are using something which creates class files at runtime (a scripting language like Groovy, for example), that will not work: In normal Java programs, classes can never be GC'd. If you still do it, the GC will run and run and achieve nothing but burn all CPU power.

Kiddy

Answer 2

Perhaps it's the same problem addressed by 1 and 2. Try to create only small transactions.

Escrow

Answer 3

I'd recommend using jstat to monitor tomcat's memory and garbage collection patterns.

You say you're facing perm gen errors, but haven't increased the amount of perm gen. Reloading tomcat apps can cause perm gen leakage, but you're probably observing your usage before you open all your pages, resulting in loading more classes into perm gen.

Slope

Thanks for the surprisingly quick replies! Yes i am using top, i am also using jstat to monitor the gc, old space reaches about 90% and then empties in about 10 secs to reach 67%. The young generations (sq,s2,e) is a continous process where in they fill and empty at regular intervals. have not increased permgen, because i thought that eventually that would fill too in time and cause even a larger old space collection pause. Is this an incorrect way of thinking? Could be locked thread am running jstack to check the thread dump, as runing jmap to see memory dump is not showing any obvious errors – Ozalid

Like to specify this is the error i get when java does its collection thing "java.lang.OutOfMemoryError" so i guess jstat is reporting incorrect figures? – Ozalid

Objects do not migrate from old space to perm gen. Perm gen is primarily for compiled classes. See blogs.sun.com/jonthecollector/entry/… – Slope

... and classes get created by deploying the application, Tomcat's JSP compiler and some persistence tools like Hibernate that use magically extended classes. Don't know about toplink or struts... – Postilion

you sir are absolutely correct, perm gen is not a part of the heap, what i meant to say is that even if i do increase the heap size, the memory leak would just fill the available space and then cause the out of memory error to occur. restarting the tomcat every three to five days, is causing a lot of stress at the users and management level, any suggestions would be welcome, will try to increase the xmx xms perm tomorrow and observe performance – Ozalid

export JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=448m -XX:MaxNewSize=448m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+CMSParallelRemarkEnabled -Djava.awt.headless=true" – Ozalid

Answer 4

your problem is that you are having in code something like this:

       new Thread(){
          public void run(){
             //something
          }
       }.start();

which in turn creates each time new class definition file in Tomcat's cache - and that eats space, so you have PermGenSpace exception...

rewrite some of your critical classes not to create custom classes 'on the fly', make separate inner classes and things will change...

      class MyJobDoer extends Thread{
          public void run(){
             //something
          }
       }

      MyJobDoer mjd=new MyJobDoer();
      mjd.start();

Intestate

This will not change anything: the compiler does just that. No class is created 'on the fly'. – Serra

we faced the same problem with permGenSpace, and it disappeared when we rewrited code this way. – Intestate

Well your problem disappeared for some other reason. – Serra

This doesn't create a new class on each execution, it creates a new anonymous class once at compile time. – Delrio

well, mine answer was written in good faith to share my experience and 'pull' out some similar experiences. Can't be sure of anything. Our Tomcat has to be restarted on every second redeploy. I think if you have class written this way tomcat might overload its perm gen space, and this is some kind of bug even people in Apache are not aware of... – Intestate