after upgrade to Spring Boot 2, how to expose cache metrics to prometheus?

Question

I recently upgraded a spring boot application from 1.5 to 2.0.1. I also migrated the prometheus integration to the new actuator approach using micrometer. Most things work now - including some custom counters and gauges.

I noted the new prometheus endpoint /actuator/prometheus does no longer publish the spring cache metrics (size and hit ratio).

The only thing I could find was this issue and its related commit.

Still I can't get cache metrics on the prometheus export. I tried settings some properties:

management.metrics.cache.instrument-cache=true
spring.cache.cache-names=cache1Name,cache2Name...

But nothing really works. I can see the Hazelcast cache manager starting up, registering the cache manager bean and so on - but neither /metrics nor /prometheus show any statistics. The caches are populated using the @Cacheable annotation. This worked with Spring Boot 1.5 - I think via Hazelcast exposing its metrics via JMX and the prometheus exporter picking it up from there?

Not sure now how to wire this together. Any hints are welcome!

Gripper

Have you read the dedicated section of the documentation? Setting the cache names if you're using Hazelcast is wrong since the purpose of that property is to create caches for you. If you have a minimal sample that reproduces the problem that I can run myself, I am happy to have a look to it. – Sorority

hello, thanks for the reply! Yes saw that section but I have trouble understanding at what point the CacheMetricsRegistrarConfiguration is running (I assume very early on) but my caches get created through scheduled tasks later on so the method bindCacheManagerToRegistry does not see any caches. Setting the property "spring.cache.cache-names" was just a desperate experiment. I'll try to strip down the application to demonstrate this. – Gripper

No need to, your case has an explicit section in the doc already. I've added an answer. – Sorority

I feared that would be the answer. I figured the cache had to be known earlier and its name start with "cache" - I non the less created that stripped down version: github.com/wemu/mobitor-parent-cache-example/blob/… - but I was not sure why. Could the registration be moved into a CacheManagerWrapper or something like that? or does that cause other implications? MY caches are actually part of plugins and currently I have no complete list of them. – Gripper

Answer 1

As you've answered my question, I can provide an answer for this.

my caches get created through scheduled tasks later on

Then this section of the doc applies to you:

Only caches that are available on startup are bound to the registry. For caches created on-the-fly or programmatically after the startup phase, an explicit registration is required. A CacheMetricsRegistrar bean is made available to make that process easier.

So you have to register such caches yourself, hopefully it is pretty easy, something like:

public class MyComponent {

    private final CacheMetricsRegistrar cacheMetricsRegistrar;
    private final CacheManager cacheManager

    public MyComponent(CacheMetricsRegistrar cacheMetricsRegistrar,
                CacheManager cacheManager) { ... }

    public void register() {
         // you have just registered cache "xyz"
         Cache xyz = this.cacheManager.getCache("xyz");
         this.cacheMetricsRegistrar.bindCacheToRegistry(xyz);
    }

}

you can include this code in your existing code. If you don't want to do that, then you need something else that runs after your existing code to register those caches to the registry.

Sorority

I'm getting this... for some reason. The following candidates were found but could not be injected: - Bean method 'cacheMetricsRegistrar' in 'CacheMetricsRegistrarConfiguration' not loaded because @ConditionalOnBean (types: org.springframework.boot.actuate.metrics.cache.CacheMeterBinderProvider,io.micrometer.core.instrument.MeterRegistry; SearchStrategy: all) did not find any beans of type org.springframework.boot.actuate.metrics.cache.CacheMeterBinderProvider – Catholicize

I'm getting the same error as @Catholicize CacheMeterBinderProvider bean is not found, and so not loaded into context. – Enyo

Followed the proposed solution and now cache_size metric is populated but cache_puts_total and cache_gets_total remain with zero. Any idea why? – Bloody

Answer 2

https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.supported.cache

Only caches that are configured on startup are bound to the registry. For caches not defined in the cache’s configuration, such as caches created on the fly or programmatically after the startup phase, an explicit registration is required. A CacheMetricsRegistrar bean is made available to make that process easier.

the key is how to get the CacheMetricsRegistrar?

if you use CaffineCache, first create a CacheMetricsRegistrar Bean in the config class like the following code

@Configuration
public class AppConfig {
    @Autowired
    private MeterRegistry meterRegistry;
    @Bean
    public CacheMetricsRegistrar cacheMetricsRegistrar() {
        return new CacheMetricsRegistrar(meterRegistry, Lists.newArrayList(new CaffeineCacheMeterBinderProvider()));
    }
}

and then the component class code like this according to your need

// inject the cacheMetricsRegistrar
@Autowired
private CacheMetricsRegistrar cacheMetricsRegistrar;

private final LoadingCache<String, String> MY_CACHE = Caffeine
            .newBuilder()
            .recordStats()
            .expireAfterAccess(10, TimeUnit.MINUTES)
            .maximumSize(100000)
            .build(this::getFromRedis);

@PostConstruct
public void init() {
        // LoadingCache -> Cache
        Cache cache = MY_CACHE;
        // bindCacheToRegistry
        cacheMetricsRegistrar.bindCacheToRegistry(new CaffeineCache("MY_CACHE", cache), Tag.of("name", "MY_CACHE"));

}

Kimura