Memory allocation behaviour with Java 1.8 in Tomcat 6 and Tomcat 8

Question

Related question: Garbage collector usage when upgrade from Java 6 + Tomcat 6 to Java 8 + Tomcat 8

I have a set of webapps, compiled with Java 8. If I run them in Tomcat 8, I get a lot of minor GC collections with a random memory allocation. In Tomcat 6 memory allocation is more linear and stable (idle in both cases, no traffic).

Eden Space Tomcat 8:

Eden Space Tomcat 6:

Do you know why that happens?

EDIT 1:

These are the data from production environment with jdk 1.8 and Tomcat 8. CPU is really high almost always due to GC cycles. Any comments about that?

EDIT 2:

This is a heapdump analisis (1.8GB dump):

Munro

With more GC calls you use less cpu. – Ettore

@GilianJoosen every GC cycle requires CPU (see cpu usage image), a memory leak could lead a continous garbage collection with a 100% of CPU time and an unusable server. – Munro

With 2 major versions between Tomcat 6 and Tomcat 8, the answer would be something like "because they changed things". What happens if you have traffic, and Tomcat isn't just sitting idle (since that's what really matters). Are you running out of memory? Does your performance degrade with Tomcat 8? – Transonic

@Transonic 5 GC Cycles per minute using 20% CPU time at idle, compared to 0 GC cycles and almost 0 % CPU sage is a performance degradation for me. I'm making Jmeter test to see what happens with higher loads – Munro

@FranMontero It's not performance degradation when running idle, unless you intend to run empty Tomcats around for obscure business reasons. If you get this behaviour running idle Tomcats, and suddenly the improvements in Tomcat 8 increase your app's throughput by 20%, you probably wouldn't be disappointed, would you? – Transonic

Ok i get your point @Kayaman. I'll update my question when i have Jmeter results in high load ;) – Munro

@Transonic Updated my question with production data – Munro

The most obvious next step is to profile both your classes and Tomcat's classes separately to see what garbage is being generated, then finding out what's generating the garbage. – Transonic

@Transonic made heapdump when memory was 1,8Gb (live and dead objects), then opened it with memory analyzer and result was that only 551MB were live objects. Updated question with image. – Munro

Keep going. Next you need to figure out what the objects are. – Nickelous

Answer 1

This is eden space, not the tenured space. So, that alone is good news.

But that step of memory appears to be tomcat8 allocating something right away after a young GC. Could it be some 'balloon' technique? (allocating a large weakly referenced buffer to 'deflate' rapidly to ensure some room in case of memory pressure). It may hide in NIO connectors too, as in the 'oomParachute' parameter (1 MB by default, but is it per httpprocessor thread? If you had 200 min threads, that would match the 200 MB seen).

I will only suggest that you can drill into the changelog to find new thing or changes that you might think they implemented wastefully like such balloon mechanism.

ALSO: you should run the tomcat6 in your jdk8 to see if it is really tomcat8 at fault. The eden space could be made larger, just in case the G1GC is so aggressive that it feels obligated to GC when a mere 200MB is used.

Weakly

Thanks for your answer. As i said in the post, i run the same apps on tomcat 6 and tomcat 8. The "extrange" behaviour only occurs with tomcat 8. I've found the problem using Javosize, look at my answer. – Munro

Answer 2

Thank you all guys for your time. In the end, the problem is a bug in tomcat 8.0_23. Using Javosize, i saw a thread consuming almost all CPU time:

Looking on the internet i found this Tomcat AJP Bug

In adittion, i've tested tomcat 8.0_32 and that Poller thread does not even exist so problem solved.

Regards

Munro