How to provide label_values in grafana variables with time range for prometheus data source?

Question

I have used a variable in grafana which looks like this: label_values(some_metric, service)

If the metric is not emitted by the data source at the current time the variable values are not available for the charts. The variable in my case is the release name and all the charts of grafana are dependent on this variable.

After the server I was monitoring crashed, this metric is not emitted. Even if I set a time range to match the time when metric was emitted, it has no impact as the query for the variable is not taking the time range into account.

In Prometheus I can see the values for the metric using the query: some_metric[24h]

In grafana this is invalid: label_values(some_metric[24h], service)

Also as per the documentation its invalid to provide $__range etc for label_values.

If I have to use the query_result instead how do I write the above invalid grafana query in correct way so that I get the same result as label_values? Is there any other way to do this?

The data source is Prometheus.

Zonate

Answer 1

I'd suggest query_result(count by (somelabel)(count_over_time(some_metric[$__range]))) and then use regular expressions to extract out the label value you want.

That I'm using count here isn't too important, it's more that I'm using an over_time function and then aggregating.

Touchhole

This worked fine ... Only point to note is the version of grafana needs to be 5.3 and above – Zonate

Use of the function $__range needs 5.3, with older version, you can use the query_result trick with a fix time range ([24h], [1w] ...) – Landowska

@Touchhole I really owe you one. Thank you very much!!! It works on Grafana 10.3 – Gassman

Answer 2

The most straightforward and lightweight solution is to use last_over_time function. For example, the following Grafana query template would return all the unique service label values for all the some_metric time series, which were available during the last 24 hours:

label_values(last_over_time(some_metric[24h]), service)

Danette

This does not work. query does not return anything. using grafana 9.1.1 and prometheus v2.38.0. I am using this in grafana dashboard variables. any idea why? – Volgograd