How to merge zero values (vector(0) with metric values in PromQL

Question

I'm using flexlm_exporter to export my license usage to Prometheus and from Prometheus to custom service (Not Grafana).

As you know Prometheus hides missing values.

However, I need those missing values in my metric values, therefore I added to my prom query or vector(0)

For example:

flexlm_feature_used_users{app="vendor_lic-server01",name="Temp"} or vector(0)

This query adds a empty metric with zero values.

My question is if there's a way to merge the zero vector with each metric values?

Edit:

I need grouping, at least for a user and name labels, so vector(0) is probably not the best option here?

I tried multiple solutions in different StackOverflow threads, however, nothing works.

Please assist.

Clipclop

Answer 1

It would help if you used Absent with labels to convert the value from 1 to zero, use clamp_max

( Metrics{label=“a”} OR clamp_max(absent(notExists{label=“a”}),0))
+
( Metrics2{label=“a”} OR clamp_max(absent(notExists{label=“a”}),0)

Vector(0) has no label.

clamp_max(Absent(notExists{label=“a”},0) is 0 with label.

Representative

Could this be leveraged to tackle multiple time series at the same time? I.e. if Metrics{label=“a”} returns multiple time series. – Aleen

Answer 2

If you do sum(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp"} or vector(0)) you should get what you're looking for, but you'll lose possibility to do group by, since vector(0) doesn't have any labels.

Maulstick

You right, but I need the grouping. So any other solution probably without vector(0), since I have to get at least user and name labels – Clipclop

I think you should be able to manually add them with recording rules, but this could be painful, if the list of values is long and/or not fixed. But then it's better to find a way for metric to always contain a value, even when 0. – Maulstick

Answer 3

I needed a similar thing, and ended up flattening the options. What worked for me was something like:

(sum by xyz(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp1"} + sum by xyz(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp2"}) or

sum by xyz(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp1"} or

sum by xyz(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp2"}

Camenae

Answer 4

There is no an easy generic way to fill gaps in returned time series with zeroes in Prometheus. But this can be easily done via default operator in VictoriaMetrics:

flexlm_feature_used_users{app="vendor_lic-server01",name="Temp"} default 0

The q default N fills gaps with the given default value N per each time series returned from q. See more details in MetricsQL docs.

Understate

Answer 5

• The VictoriaMetrics approach does not work for prometheus
• The "clamp_max" approach does only keep labels that are known already
• Other approaches seem quite complicated

Reusing parts of the series and multiplying them with 0 will instead effectively result in a vector(0) with all proper labels of the original series. Just put the vectors into a min aggregation and you'll get a proper default value of 0 (assuming only positive values of course).

In the examples the existence of the Grafana variable $__interval is assumed - replace it with any specific interval if needed.

General Example showing the advantage of keeping all labels:

min(series{label="a"} or 0 * min_over_time(series{label="a"}[$__interval])) by (anotherLabel)

Specific Example to answer the question:

min(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp"} or 0 * min_over_time(flexlm_feature_used_users{app="vendor_lic-server01",name="Temp"}[$__interval]))

Herzberg