Micrometer - Prometheus Gauge displays NaN

Question

I am trying to generate Prometheus metrics with using Micrometer.io with Spring Boot 2.0.0.RELEASE.

When I am trying to expose the size of a List as Gauge, it keeps displaying NaN. In the documentation it says that;

It is your responsibility to hold a strong reference to the state object that you are measuring with a Gauge.

I have tried some different ways but I could not solve the problem. Here is my code with some trials.

import io.micrometer.core.instrument.*;
import io.swagger.backend.model.Product;
import io.swagger.backend.service.ProductService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

@RestController
@RequestMapping("metrics")
public class ExampleController {

    private AtomicInteger atomicInteger = new AtomicInteger();

    private ProductService productService;
    private final Gauge productGauge;

    @Autowired
    public HelloController(ProductService productService,
                           MeterRegistry registry) {

        this.productService = productService;

        createGauge("product_gauge", productService.getProducts(), registry);
    }

    private void createGauge(String metricName, List<Product> products,
                                    MeterRegistry registry) {

        List<Product> products = productService.getProducts();

        // #1
        // this displays product_gauge as NaN
        AtomicInteger n = registry.gauge("product_gauge", new AtomicInteger(0));
        n.set(1);
        n.set(2);

        // #2
        // this also displays product_gauge as NaN
        Gauge
            .builder("product_gauge", products, List::size)
            .register(registry);

        // #3
        // this displays also NaN
        testListReference = Arrays.asList(1, 2);
        Gauge
            .builder("random_gauge", testListReference, List::size)
            .register(registry);

        // #4
        // this also displays NaN
        AtomicInteger currentHttpRequests = registry.gauge("current.http.requests", new AtomicInteger(0));
    }

    @GetMapping(path = "/product/decrement")
    public Counter decrementAndGetProductCounter() {
        // decrement the gague by one
    }
}

Is there anyone who can help with this issue? Any help would be appreciated.

Ponce

Answer 1

In all cases, you must hold a strong reference to the observed instance. When your createGauge() method is exited, all function stack allocated references are eligible for garbage collection.

For #1, pass your atomicInteger field like this: registry.gauge("my_ai", atomicInteger);. Then increment/decrement as you wish. Whenever micrometer needs to query it, it will as long as it finds the reference.

For #2, pass your productService field and a lambda. Basically whenever the gauge is queried, it will call that lambda with the provided object: registry.gauge("product_gauge", productService, productService -> productService.getProducts().size());

(No guarantee regarding syntax errors.)

Lemar

I am doubtful that the second solution will work. The productService that is being passed to the guage method will still be referred with a WeekReference and hence will be garbage collected. Meaning when the lambda will be called and the parameter is required to be passed it will be null. This possibly will fail the lambda execution rather than reporting NaN. – Inaugural

As we are talking about a Spring managed Bean there, it will work as long as the context is around. – Lemar

My bad your example code is correct. In this scenario, WeekReference will not be an issue, since it is pointing to a bean which will not be garbage collected. I just want to point out the reason it is working is that it's using an object which will remain in memory due to Hard reference from the applicationContext. And hence will not be garbage collected. The following might not work: ProductService productService = ... // defined in local context. registry.gauge("product_gauge", productService, productService -> productService.getProducts().size()); – Inaugural

Answer 2

I wasn't able to use @panser solution 'cause I'm using gauges with labels. My solution involved the creation of com.google.common.util.concurrent.AtomicDouble cache with io.micrometer.core.instrument.Tag's key and values as map key, heres goes:

    private static final Map<String, AtomicDouble> GAUGE_CACHE = new HashMap<>();

    public void handleGauge(String name, List<Tag> tags, double value) {
        String gaugeKey = this.gaugeKey(name, tags);
        if (!GAUGE_CACHE.containsKey(gaugeKey)) {
            GAUGE_CACHE.put(gaugeKey, new AtomicDouble());
        }
        Objects.requireNonNull(this.registry.gauge(name, tags, GAUGE_CACHE.get(gaugeKey))).set(value);
    }

    private String gaugeKey(String name, List<Tag> tags) {
        return name + ":" + tags.stream().map(tag -> tag.getKey() + tag.getValue()).collect(Collectors.joining(""));
    }

That worked pretty well for my needs, hopefully help other people.

Prisca

Answer 3

I had the same issue with Micrometer.io gauges when I used your method #1 meterRegistry.gauge("myGauge", new AtomicDouble()). I am using Scala by the way. I noticed that after I created about 50 gauges, the new gauges after that displayed NaN.

Instead I used:

val atomicDouble = new AtomicDouble()
Gauge
  .builder("myGauge", atomicDouble, new AtomicDoubleToDoubleFunction)
  .strongReference(true)
  .register(meterRegistry)

with

class AtomicDoubleToDoubleFunction extends ToDoubleFunction[AtomicDouble] {
  override def applyAsDouble(value: AtomicDouble): Double = value.doubleValue()
}

This fixed the NaN issue, and all of my gauges appear correctly. I found the .strongReference(true) example from https://www.codota.com/code/java/classes/io.micrometer.core.instrument.Gauge .

Winnebago

Either way you will still need to hold the reference to the AtomicDouble you create, using the builder doesn't change much in this regard. If you attempt to call the builder twice with an updated gauge value it will do nothing (the gauge will stay at the previous value forever). – Seddon

Answer 4

my example for gauge using

private final AtomicLong countTryUsers = new AtomicLong(0);
Metrics.gauge("app.countTry", countTryUsers);

public void updateCountTryUsers(Long countTryUsersDb){
   countTryUsers.set(countTryUsersDb);
}

so I register app.countTry just once, and then just update AtomicLong countTryUsers over custom method updateCountTryUsers()

Kriss

Answer 5

I was getting the same issue, in spring boot with actuator and prometheus, where I was trying to create a metric for Map size. When I register the map with the guageMapSize, for some time it was showing the correct value, and after that it just stopped updating, and continuously showed NAN, as the metric for map with key already present, it was not updating.

To resolve this we can remove the existing metric and register the new one with the updated value. This worked for me. Hope it works for you.

Create a method, removeMicroMeterGuage(); use it to remove a gauge, and then add new gauge with the updated value.

removeMicroMeterGuage(<keyName>);
registry.gaugeMapSize(<KeyName>, null, map);



 public void removeMicroMeterGuage(String guageName) {
        List<Meter> meterList = registry.getMeters();
        for(Meter meter : meterList){
            meter.getId().getName();
            if(meter.getId().getName().equalsIgnoreCase(guageName)){
                System.out.println(guageName+" guage removed");
                registry.remove(meter);
            }
        }
    }

Adhesion

Answer 6

From the docs:

10.3. Why is My Gauge Reporting NaN or Disappearing?

It is your responsibility to hold a strong reference to the state object that you are measuring with a Gauge. Micrometer is careful to not create strong references to objects that would otherwise be garbage collected. Once the object being gauged is de-referenced and is garbage collected, Micrometer starts reporting a NaN or nothing for a gauge, depending on the registry implementation.

If you see your gauge reporting for a few minutes and then disappearing or reporting NaN, it almost certainly suggests that the underlying object being gauged has been garbage collected.

Valaria