Java G1: Monitoring for memory leaks in production

Question

For years, we've been running Java services with modest heap sizes using +UseParallelOldGC. Now, we're starting to roll out a new service using a larger heap and the G1 collector. This is going pretty well.

For our services that use +UseParallelOldGC, we monitor for memory leaks by looking at the old generation size after collection and alerting on a threshold. This works quite well, and in fact saved our bacon just two weeks ago.

Specifically, for +UseParallelOldGC, we do the following:

• ManagementFactory.getMemoryPoolMXBeans()
• Search for the MemoryPoolMXBean result with the name ending in "Old Gen"
• Compare getCollectionUsage().getUsed() (if available) with getMax()

Unfortunately, it seems like G1 no longer has a concept of getCollectionUsage().

Fundamentally, though, we'd like to monitor the G1 heap size following the last mixed collection it chooses to do in a mixed cycle, or something similar.

For example, outside the VM I would be happy with an awk script that merely found the last '(mixed)' was that's followed by a '(young)' and look what the final heap size was (e.g., '1540.0M' 'Heap: 3694.5M(9216.0M)->1540.0M(9216.0M)')

Is there any way to do this inside the Java VM?

Satirize

this should provide similar information: https://mcmap.net/q/821276/-monitoring-cpu-ram-i-o-usage-at-time-of-java-garbage-collection – Prepense

Answer 1

Yes, JVM gives you enough tools to retrieve such information for G1. For instance, you could use something like this class that prints all the details about garbage collections (just call MemoryUtil.startGCMonitor()):

public class MemoryUtil {

    private static final Set<String> heapRegions;

    static {
        heapRegions = ManagementFactory.getMemoryPoolMXBeans().stream()
                .filter(b -> b.getType() == MemoryType.HEAP)
                .map(MemoryPoolMXBean::getName)
                .collect(Collectors.toSet());
    }

    private static NotificationListener gcHandler = (notification, handback) -> {
        if (notification.getType().equals(GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION)) {
            GarbageCollectionNotificationInfo gcInfo = GarbageCollectionNotificationInfo.from((CompositeData) notification.getUserData());
            Map<String, MemoryUsage> memBefore = gcInfo.getGcInfo().getMemoryUsageBeforeGc();
            Map<String, MemoryUsage> memAfter = gcInfo.getGcInfo().getMemoryUsageAfterGc();
            StringBuilder sb = new StringBuilder(250);
            sb.append("[").append(gcInfo.getGcAction()).append(" / ").append(gcInfo.getGcCause())
                    .append(" / ").append(gcInfo.getGcName()).append(" / (");
            appendMemUsage(sb, memBefore);
            sb.append(") -> (");
            appendMemUsage(sb, memAfter);
            sb.append("), ").append(gcInfo.getGcInfo().getDuration()).append(" ms]");
            System.out.println(sb.toString());
        }
    };

    public static void startGCMonitor() {
        for(GarbageCollectorMXBean mBean: ManagementFactory.getGarbageCollectorMXBeans()) {
            ((NotificationEmitter) mBean).addNotificationListener(gcHandler, null, null);
        }
    }

    public static void stopGCMonitor() {
        for(GarbageCollectorMXBean mBean: ManagementFactory.getGarbageCollectorMXBeans()) {
            try {
                ((NotificationEmitter) mBean).removeNotificationListener(gcHandler);
            } catch(ListenerNotFoundException e) {
                // Do nothing
            }
        }
    }

    private static void appendMemUsage(StringBuilder sb, Map<String, MemoryUsage> memUsage) {
        memUsage.entrySet().forEach((entry) -> {
            if (heapRegions.contains(entry.getKey())) {
                sb.append(entry.getKey()).append(" used=").append(entry.getValue().getUsed() >> 10).append("K; ");
            }
        });
    }
}

In this code, gcInfo.getGcAction() gives you enough information to separate minor collections from major/mixed ones.

But there's an important caveat to using your approach (with a threshold) to G1. A single mixed collection in G1 usually affects only several old gen regions - many enough to free sufficient amount of memory but not too many in order to keep the GC pause low. So, after a mixed collection in G1 you cannot be sure that all your garbage has gone. As a result, you need to find more sophisticated strategy to detect memory leaks (maybe based on collections frequency, gathering statistics from several collections, etc.)

Clement