Garbage Collection Overhead Help Needed

Question

I am looking for a little bit of guidance in trying to diagnose a GC related issue.

We are testing on Solaris with WebSphere Portal and my current environment has a Garbage Collection Overhead of 7% (This was calculated with verbose GC and running the log through PMAT) I am supposed to compare this value with another environment which is running at an avg of 4.5%. The env are on the exact same version of WebSphere Portal, same JVM sizes,parameters,custom varibles,etc. My JVMs had 20 more Allocation Failures, 2 more full GC's, a 2 sec higher mean pause time during GCs than their environment during a 1 hr performance testing period.

Can you give me any advice on what could be causing this issue with all the same configuration values and the same exact 1 hr performance test? Or anything else to review?

Thanks

Platas

Are these two machines running in a 100% fair load balancing environment? Same virtual machines and same hardware? – Whitecollar

Yes, they are running in a 100% fair load balancing environment. Same hardware, non-virtualized. I actually found a pretty big difference, when they ran their test they had JVM settings which included: -XX:ParallelGCThreads=32 they then changed this to -XX:ParallelGCThreads=8 at a later time. We set our env equal to 8 to match their environment not knowing it was set to 32 during the test that we were comparing to. Could this alone explain the difference? – Platas

Frankly, 7% overhead is damn low. (Though of course the Sun JVM seriously understates GC overhead, but that's another matter.) – Overblouse

Thanks Daniel, I agree with you but mgmt is asking why is it 2% above the other environment numbers. I also confirmed that the server has 32 cpus so by setting our env XX:ParallelGCThreads=8 might explain the difference vs their env that was set to XX:ParallelGCThreads=32. Anyone think that could cause the extra 2% overhead? – Platas

Possibly, as you would be using less cores to perform GC. – Whitecollar

Do you have an option to capture and analyze heap dumps of these two WPS instances? Looking at the objects occupying your memory may give you a hint at what is different between the two environments. – Redaredact

Yes it can make the difference in a stress test. Stress testing you can easily exaggerate effects of different gc settings. And that shows that it is worth to optimize GC if you cannot scale anymore or, even better, to scale out. In the experience of the web portal I am working for GC settings always used to have a huge impact and can make the difference between unoperable and perfectly healthy. So, now you can answer the question yourself and thus close it ;-) – Lancashire

How does the hardware compare on the 2 systems? Is it the same? – Judicial

Are you sure that the jobs running on the two different machines were insignificant to influence the results? – Grandfatherly

Could you confirm that setting those two environments to exactly same configurations (XX:ParallelGCThreads) makes the results the same? – Ultramicroscope

Answer 1

The first thing I'd check is whether the process is ever being pushed into swap / "virtual" memory. This would have a significant impact on your GC overhead.

With identical Java configurations and load etc., the most obviously difference would be the amount of physical RAM on each machine. Assuming they're the same, see what other processes / services are running on each machine - unless they're both fresh installs of the same OS, I'd expect there to be some (potentially significant) differences.

Post the output from top and iostat (or your preferred tools) and we'll see if there's anything obvious.

Cowherb

Answer 2

If you are running different Operating systems it's possible that they perform differently with same settings. The optimal GC settings and threadpool-sizes might be different. I remember that at least in Solaris vs RedHat case the optimal concurrency-settings were clearly different.

Cadge