High memory usage issues with G1 Garbage collector

Question

We have been testing out G1 garbage collector recently with the following configuration:

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseG1GC -XX:MaxGCPauseMillis=1250 -XX:+PrintTenuringDistribution -Xloggc:${logdir}/gc-$(date +%Y_%m_%d-%H_%M).log -XX:+UseStringDeduplication -XX:+PrintStringDeduplicationStatistics -XX:+PrintPromotionFailure -XX:+PrintAdaptiveSizePolicy -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:ParallelGCThreads=8 -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8M JAVA_OPTS_HEAP: -Xms16g -Xmx16g

We had recently come across an issue recently where two java processes were running with the above configuration on a box with 48 GB RAM and both the processes went on to consume around 20 - 22 GB of RAM each (few small processes consuming the remaining memory), thus filling the entire RAM and then it triggered disk swaps which finally led to the OOM and process getting killed.

This seems worrying because neither the NMT reports this memory usage in a meaningful way nor do we get any clues for this usage from the GC logs. In the NMT stats, the application memory was under 16G and the metaspace usage was under 1G.

We had tried setting maxMetaSpaceSize to 2G but that didn't help either. The RAM usage seems to grow unbounded when the process is running for days.

From the other questions it does seem that G1 garbage collector does tend to consume more memory but the disk swaps is a worrying issue. Could someone please provide some pointers on how this issue could be resolved?

Lancelot

any updates on this issue – Grub

No we haven't found the root cause for this as yet. – Lancelot

We've encounter a similar situation. System runs fine for several days. 15% heap consumption, 30% total RAM consumption. Then, over a period of a few minutes, the JVM proceeds to consume all available RAM (10G total) and then start swapping. The heap remains the same, but the monitoring goes wonky as the system is unresponsive since it's just paging like a mad man. It's production traffic, and then one day, just ZZzzang! To the moon, Alice! We're switching back, see what happens in a few days. – Piezochemistry

Answer 1

As to long for a comment I put it in as an answer.

A good reading which explains why a java process might consume more memory than -Xmx. Based ony our provided information I believe this would be also the reason in your case.

For G1 there is an OBE Getting Started with the G1 Garbage Collector with details about the function of the G1GC. Have a look there for Recommended Use Cases for G1. Maybe you would not benefit from using G1.

cited from from the OBE (Oracle By Example)

If you are using CMS or ParallelOldGC and your application is not experiencing long garbage collection pauses, it is fine to stay with your current collector.

Galley

We have definitely got performance benefit using G1. We have got better throughput and we don't see full gc happening which almost used to happen daily with CMS because of heap getting heavily fragmented. G1 does suit our needs based on the testing. – Lancelot

@AnishGupta Then you need to analyze the GC log file (here's an older post which might help in the beginning blogs.oracle.com/poonam/understanding-g1-gc-logs) to see if you might tune it to consume less memory, without degrading the better throughput or add more RAM (if possible). – Galley

Answer 2

Here you can find testing results of G1, Parallel, ConcMarkSweep, Serial and Shenandoah garbage collectors in terms of scaling and resource consumption, as well as some suggestions on what settings can be applied to improve results. So you can choose the most appropriate one for your project and reduce memory usage.

Wroth