Springboot 2 micrometer-registry-prometheus How to create your own histogram buckets

Question

Hi I was trying to create histogram buckets but in Springboot 2 is it possible to create custom buckets or equal interval buckets something like below

le="0.005", le="0.01" , le="0.025" , le="0.05", le="0.075" ...  le="+Inf"

Like:

XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.005"**,} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.01",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.025",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.05",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.075",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.1",} 2.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.25",} 2.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.5",} 2.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.75",} 2.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="1.0"**,} 2.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="2.5"**,} 2.0

Instead of Currently output of /actuator/prometheus looks like :

XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.001048576**",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.001398101**",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.001747626**",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.002097151**",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",**le="0.002446676**",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.002796201",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.003145726",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.003495251",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.003844776",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.004194304",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.005592405",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.006990506",} 0.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="0.008388607",} 0.0
:
cket{APP="XYZ",class="HelloController",exception="none",method="hello",le="22.906492245",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="28.633115306",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="30.0",} 1.0
XYZ_seconds_bucket{APP="XYZ",class="HelloController",exception="none",method="hello",le="+Inf",} 1.0
XYZ_seconds_count{APP="XYZ",class="HelloController",exception="none",method="hello",} 1.0

Configuration Class looks like

@Configuration
@EnableAutoConfiguration
@ComponentScan
class TimingConfiguration {
  @Bean
  fun timedAspect(registry: MeterRegistry?): TimedAspect {
    return TimedAspect(registry!!)
  }
}

Controller has annotated function :

@Timed(description = "Get hello", histogram = true)
@GetMapping(\hello)

build.gradle has following dependencies:

implementation("org.springframework.boot:spring-boot-starter")
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")
implementation("org.springframework.boot:spring-boot-starter-aop3")

Portia

Please edit you question to more readable and understandable format. – Corbie

Would this help baeldung.com/micrometer#5-distributionsummary ? – Scribble

Answer 1

tl;dr:

• turn off percentile histograms
• add any number of SLA boundaries

See this (closed) issue: https://github.com/micrometer-metrics/micrometer/issues/1947

You can control the buckets added by @Timed with a MeterFilter that implements the configure method. Some shortcuts:

• MeterFilter.maxExpected("http.server.requests", Duration.ofSeconds(1))
• MeterFilter.minExpected("http.server.reqests", Duration.ofMillis(10))

MeterFilter implementations simply need to be wired as a @Bean in a Spring app to take effect.

Even more of a shortcut, Spring boot has property-driven configuration for setting minimum and maximum expected values:

https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#per-meter-properties

Also, there is no need to use @Timed on Spring MVC and WebFlux endpoints, as these are automatically instrumented by the framework.

In general we try to discourage fine grained control of histogram buckets because experience has shown that folks tend to select buckets leading to high-error-bound percentile approximations (and the true error bound is in fact unknowable from the discretized distribution, so they never really know). Technically sla options just add additional bucket values. So if you really really want to fight the recommendation, you can turn off percentile histograms and add any number of SLA boundaries. This in effect yields a histogram of a defined set of bucket boundaries.

@jkschneider

[The temptation to set the number of buckets] is actually precisely why we removed (obvious) configurability. It's just too easy to say "hey it's publishing 100 buckets right now, can we turn that down to 50 buckets?" But the impact on percentile approximation is unknowable. Narrowing min/max (provided your timings don't go outside of that range) doesn't affect the approximation's accuracy.

@jkschneider

Purdy

Answer 2

# disable default percentilesHistograms
management.metrics.distribution.ph.http.server.requests=false
# set custom ServiceLevelObjectiveBoundaries
management.metrics.distribution.slo.http.server.requests=300ms,500ms,1s,2s,5s,10s,15s,20s,30sis

Tips: http.server.requests is name of the Timer metric

Veradia