How to have a metric about Java HeapSpace exceptions in an Application Server

Question

We are using an old version of Payara, and exporting JVM metrics with the Prometheus JVM GC Exporter and Memory exporters.

In these exporters, we don't have access to the number of Java HeapSpace exceptions, which is an indicator of a inadequate value of the XMX.

I have a dashboard where I survey the memory status of our Java apps, and here is a typical example of a "memory starved" app:

I could use the GC average duration, but I'd rather have a "hard fact" counter of the number of Java Heapspace exceptions, which is the real indicator of a lack of memory. Since we are operating in a Java App server, we don't have a main() to intercept all Java HeapSpace exceptions (and we won't be able to be sure that it wasn't intercepted below so the counter might be faulty anyway).

Do you guys have had the same issues, and found a way to measure this problem? I explored the JMX tree but didn't find conclusive metrics.

Promontory

@Guli13 what do you refer with "Java HeapSpace exceptions"? java.lang.OutOfMemoryError? – Guiana

Which JVM, which version ? Which OS ? Which Payara version ? – Gallia

We are using Java 8 (the last one, through Temurin) and Payara 4.174. What I want is to have metrics about the number of java.lang.OutOfMemoryError indeed! I can't install a "global try catch" on this, since it is managed by Payara. I'm wondering how to do that. – Promontory

Answer 1

Since you are already using Prometheus to monitor your application, you could use the mtail exporter to count your exceptions from the logs and expose the metrics for Prometheus to scrape.

mtail is a tool for extracting metrics from application logs to be exported into a timeseries database or timeseries calculator for alerting and dashboarding. It fills a monitoring niche by being the glue between applications that do not export their own internal state (other than via logs) and existing monitoring systems, such that system operators do not need to patch those applications to instrument them or writing custom extraction code for every such application.

Halvorson