Micrometer/Prometheus How do I keep a gauge value from becoming NaN?

Question

I am trying to monitor logged in users, i am getting the logged in user info by calling api, this is the code i have used,

public class MonitorService {
    private InfoCollectionService infoService;
    public MonitorService(InfoCollectionService infoService) {
        this.infoService = infoService
    }

    @Scheduled(fixedDelay = 5000)
    public void currentLoggedInUserMonitor() {
        infoService.getLoggedInUser("channel").forEach(channel -> {
            Metrics.gauge("LoggedInUsers.Inchannel_" + channel.getchannelName(), channel.getgetLoggedInUser());
        });
    }
}

And i see the values in Prometheus, the problem is after a few seconds, the value become NaN, i have read that Micrometer gauges wrap their obj input with a WeakReference(hence Garbage Collected ).I don't know how to fix it.If anybody knows how to fix this it would be great.

Scratches

Answer 1

This is a shortcoming in Micrometer that I would like to fix eventually.

You need to keep the value in a map in the meantime so it avoid the garbage collection. Notice how we then point the gauge at the map and us a lambda to pull out the value to avoid the garbage collection.

public class MonitorService {
    private Map<String, Integer> gaugeCache = new HashMap<>();
    private InfoCollectionService infoService;
    public MonitorService(InfoCollectionService infoService) {
        this.infoService = infoService
    }

    @Scheduled(fixedDelay = 5000)
    public void currentLoggedInUserMonitor() {
        infoService.getLoggedInUser("channel").forEach(channel -> {
            gaugeCache.put(channel.getchannelName(), channel.getgetLoggedInUser());
            Metrics.gauge("LoggedInUsers.Inchannel_" + channel.getchannelName(), gaugeCache, g -> g.get(channel.getchannelName()));
        });
    }
}

I would also recommend using tags for the various channels:

Metrics.gauge("loggedInUsers.inChannel", Tag.of("channel",channel.getchannelName()), gaugeCache, g -> g.get(channel.getchannelName()));

Cimbri

tried to use the same pattern without success. Dont understand why you pass the whole map in the gauge. – Depart

This helped me out a lot! Just to share I had to use a strong reference as per the docs "If you see your gauge reporting for a few minutes and then disappearing or reporting NaN, it almost certainly suggests that the underlying object being gauged has been garbage collected." micrometer.io/docs/… Which in my case was as simple as using an Instance Variable which fixed it immediately. – Cabal

Answer 2

You could use a newer solution like:

Gauge.builder("LoggedInUsers.Inchannel_" + channel.getchannelName(), channel.getgetLoggedInUser(), n -> n).strongReference(true).tags(tags).register(meterRegistry);

Fussell

Please read How to Answer and edit your answer to contain an explanation as to why this code would actually solve the problem at hand. Always remember that you're not only solving the problem, but are also educating the OP and any future readers of this post. – Arvind