Prometheus jmx exporter pattern match for attributes and items

Question

I am trying to pattern match and name prometheus metrics with the jmx_exporter java agent (https://github.com/prometheus/jmx_exporter).

There is not much documentation on how to pattern match on MBean attributes and items within these attributes when dealing with CompositeType.

For instance, I got to the point where I pattern in such a way:

rules:
  - pattern: "java.lang<type=Memory><>HeapMemoryUsage"
    name: jmx_jvm_memory_HeapMemoryUsed

But if you look in VisualVM at HeapMemoryUsed attribute. You can also see in the Attribute Description in openType the following:

javax.management.openmbean.CompositeType(
    name=java.lang.management.MemoryUsage,
    items=(
            (itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
            (itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
            (itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
            (itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long))
        )
    )

I want to be able to name the metric based on these items. For example, I would like to have a metrics such as:

• jmx_jvm_memory_HeapMemoryUsed_used
• jmx_jvm_memory_HeapMemoryUsed_max

etc...

Thanks!

Lasonyalasorella

Answer 1

You're right that the documentation is sparse on this topic. If you look into the source code however, you will find that beans with composite data type are unfolded into metrics for each combination of attribute and value (JMXScraper.java:207-224).

---

In your case, you can achieve the desired result with the following rules item:

rules:
  # ... other rules
  - pattern: 'java.lang<type=Memory><(HeapMemoryUsage>(\w+): (.*)'
    name: jmx_jvm_memory_HeapMemoryUsed_$1
    help: "JVM heap memory $1"
    value: $2
    type: GAUGE

If you start your server with JMXExporter agent enabled and query the metrics endpoint, you will see something like the following output

$ curl -s ${server-url}:${jmx-exporter-port}/metrics | grep jmx_jvm
# HELP jmx_jvm_memory_HeapMemoryUsed_committed JVM heap memory committed
# TYPE jmx_jvm_memory_HeapMemoryUsed_committed gauge
jmx_jvm_memory_HeapMemoryUsed_committed 7.7856768E8
# HELP jmx_jvm_memory_HeapMemoryUsed_max JVM heap memory max
# TYPE jmx_jvm_memory_HeapMemoryUsed_max gauge
jmx_jvm_memory_HeapMemoryUsed_max 1.908932608E9
# HELP jmx_jvm_memory_HeapMemoryUsed_init JVM heap memory init
# TYPE jmx_jvm_memory_HeapMemoryUsed_init gauge
jmx_jvm_memory_HeapMemoryUsed_init 2.64241152E8
# HELP jmx_jvm_memory_HeapMemoryUsed_used JVM heap memory used
# TYPE jmx_jvm_memory_HeapMemoryUsed_used gauge
jmx_jvm_memory_HeapMemoryUsed_used 4.7050592E8

---

Note: as Brian already said in his answer, there is no need to do this for standard JMX beans (such as the ones in the java.lang domain), as JMXExporter already handles them in a standardized way.

Healthful

Answer 2

If you look at the HELP in the metrics output, that's what you're matching against. However there's no need for you to worry about any of the java.lang metrics, the jmx exporter java agent will provide them for you automatically under the jvm_ metrics prefix.

Divisionism

What do you mean by HELP in the metrics output? – Lasonyalasorella

@Lasonyalasorella If you don't supply any pattern for a set of beans, the default formatter is applied. In the output of the default formatter the # HELP ... line will contain the bean pattern required by JMXExporter to match the given bean in parens. For the standard JMX beans (for example domain java.lang), you won't see this however, since they seem to have a format which is shipped with JMXExporter. In your case this should be jvm_memory_bytes_used{area="heap",} and jvm_memory_bytes_max{area="heap",}. – Healthful

Answer 3

For anyone encountering issues with this, and getting stuck. I managed to get it to work.

I'm currently porting HBase JMX metrics to Prometheus, and was figuring out how to update metrics with composite values, lets look at an example:

{
    "name": "java.lang:type=Memory",
    "modelerType": "sun.management.MemoryImpl",
    "ObjectPendingFinalizationCount": 0,
    "Verbose": false,
    "HeapMemoryUsage": {
        "committed": 127729664,
        "init": 132120576,
        "max": 2087452672,
        "used": 26782688
    },
    "NonHeapMemoryUsage": {
        "committed": 50896896,
        "init": 2555904,
        "max": -1,
        "used": 49540216
    },
    "ObjectName": "java.lang:type=Memory"
},

By default, you'll have metrics formatted as below:

# HELP java_lang_Memory_HeapMemoryUsage_init java.lang.management.MemoryUsage (java.lang<type=Memory><HeapMemoryUsage>init)
# TYPE java_lang_Memory_HeapMemoryUsage_init untyped
java_lang_Memory_HeapMemoryUsage_init 1.32120576E8

But if you're like me, you probably want it like so:

# HELP hbase_heap_usage java.lang.management.MemoryUsage (java.lang<type=Memory><HeapMemoryUsage>committed)
# TYPE hbase_heap_usage untyped
hbase_heap_usage{type="committed",} 1.27729664E8
hbase_heap_usage{type="init",} 1.32120576E8
hbase_heap_usage{type="max",} 2.087452672E9
hbase_heap_usage{type="used",} 2.8101728E7

To achieve this, the documentation is not great, and a little bit backwards, when using VisualVM, it will tell you that HeapMemoryUsage is the attribute, but in these composite cases, the keys inside the attribute are the attributes, and the attribute is the key ... so to achieve the above, your rule will look like as such:

  - pattern: java.lang<type=Memory><HeapMemoryUsage>(\w+)
    name: hbase_heap_usage
    labels:
      type: $1

Burlie