Divide two metrics in prometheus
Asked Answered
J

3

9

I have two metrics, scratched from telegraf.

first metric

vsphere_host_mem_active_average{esxhostname=~"esx1"}

gives one value

vsphere_host_mem_active_average{clustername="BCH1",collector="telegraf",dcname="DC",esxhostname="esx1",host="vm01",hostname="hostname1",instance="localhost:9273",job="vSphere",moid="host-78563",source="esx1",type="vmware",vcenter="vmc"}    17763152

second one

vsphere_vm_mem_granted_average{esxhostname=~"esx1"})

gives several with different labels

1.

vsphere_vm_mem_granted_average{clustername="BCH1",dcname="DC",esxhostname="esx1",guest="debian9_64",host="vm01",moid="vm-79139",source="vm01",uuid="42244f7b-abeb-92be-3e67-af19a9d8dfbd",vcenter="vmc",vmname="vm01"} 4.19418e+06

2.

vsphere_vm_mem_granted_average{clustername="BCH1",dcname="DC",esxhostname="esx1",guest="debian9_64",host="vm01",moid="vm-79146",source="vm01",uuid="4224ed0c-f306-202c-fc99-35e48fe52370",vcenter="vmc",vmname="vm02"} 8.377904e+06

is that possible to get result of first value divided by second values for each of set of labels?

Jd answered 21/1, 2019 at 13:6 Comment(1)
As mentioned in PromQL documentation you can just divide two instant vectors and resulting vector will contain result of division for each entries with same set of label values. Is it what you want?Tresatrescha
H
18

Prometheus applies arithmetic operators such as /, -, +, * individually per each pair of time series with identical set of labels (ignoring metric name) on both sides of the operator. If there are no pairs of time series with identical labels, then Prometheus returns nothing. See these docs for more details. This behavior can be augmented by applying on(), ignoring(), group_left() and group_right() modifiers - see these docs.

So, if you need to divide a single time series by two other time series with distinct set of labels, then the following PromQL query should work:

vsphere_host_mem_active_average{esxhostname=~"esx1"}
  / on() group_right()
vsphere_vm_mem_granted_average{esxhostname=~"esx1"}

The on() modifier instructs Prometheus to limit the set of labels, which are used to find pairs of time series with identical labels on the left and the right side of / operator, to an empty set. The empty list at on() modifier automatically matches any time series on the left side of / operator to any time series on the right side of / operator.

The group_right() modifier instructs Prometheus to divide a single time series on the left side of / operator individually to every time series on the right side of / operator. The resulting time series contain labels from the right side time series. Additional labels from time series on the left side can be added to result by enumerating them inside group_right() modifier.

Hyphen answered 29/3, 2022 at 13:45 Comment(0)
A
1

Perhaps I am not interpreting the question quite right, but if you want to divide 2 metrics together, you simply use the '/' operator in your PromQL, documented here: https://prometheus.io/docs/prometheus/latest/querying/operators/

As an example using your above values, the query you would use is something like:

(vsphere_host_mem_active_average{clustername="BCH1",collector="telegraf",dcname="DC",esxhostname="esx1",host="vm01",hostname="hostname1",instance="localhost:9273",job="vSphere",moid="host-78563",source="esx1",type="vmware",vcenter="vmc"}/vsphere_vm_mem_granted_average{clustername="BCH1",dcname="DC",esxhostname="esx1",guest="debian9_64",host="vm01",moid="vm-79139",source="vm01",uuid="42244f7b-abeb-92be-3e67-af19a9d8dfbd",vcenter="vmc",vmname="vm01"})
Ament answered 15/5, 2019 at 22:22 Comment(1)
I think the question is more specifically asking how to do this without knowing all possible label values ahead of time.Quechua
M
0

Also my answer is not really related to the OPs question I want to share my findings if anyone finds this answer with the same problem like me.

My problem was that I have two vectors with almost the same set of labels but only almost. One vector has one additional label (cpu) hence I'm not able to divide the two vectors.

The solution is actually pretty easy. You need to remove the extra label. You can do this with the label_replace function of promql.

My complete solution looks like this:

label_replace(
  rate(container_cpu_usage_seconds_total {
    namespace='namespace',
    pod=~'.+-oracle-.+',
    container=~'[a-z-]+',
    container!='sidecar'
  }[3m]),
 'cpu', '', 'cpu', '.*'
) / (
  container_spec_cpu_quota {
    namespace='namespace',
    pod=~'.+-oracle-.+',
    container=~'[a-z-]+',
    container!='sidecar'
  } / 100000
)

You can also chain the label_replace function to remove multiple labels.

For the sake of completeness: This query returns the relative cpu usage for specific containers in a k8s cluster namespace.

Multicolor answered 14/6, 2024 at 12:23 Comment(1)
To be clear, provided solution will only work if other labels identify each time series uniquely. If not, you'd need to use aggregation, for example sum without(cpu) (<your_metric_here>)Yulma

© 2022 - 2025 — McMap. All rights reserved.