Prometheus/Grafana Rate()....what Unit for Y Axis

Question

I have a counter that I am plotting on Grafana.

rate(processed_work_items_total{job="MainWorker"}[1m])

I am not getting the expected numbers in Grafana.

What I want is the # of Work Items Processed per minute.

Is my query wrong? or my Unit of Measure in my Y Axis. I currently have it as ops/min and its giving me a super small number.

Mayday

After a bit of google...maybe my query should be sum(increase(processed_work_items_total[1m]))? Is that correct? If so then what is the Unit of measure for my Y Axis. – Mayday

Answer 1

According to the documentation, rate(processed_work_items_total{job="MainWorker"}[1m]) will calculate the number of work items processed per second, measured over the last one minute (that's the [1m] from your query).

If you want the number of items per minute, simply multiply the above metric with 60.

Moneywort

Thanks Oliver. Should I decrease the [1m] to something like [15s], then multiply that by 4, to get a more granular graph? – Mayday

No, the [1m] refers to over what time frame the rate is measured but the rate itself is always in items per second. You could use irate (see the docs for an explanation of rate vs irate) if you have very spiky data but generally you should still multiply by 60 if you want item per minute. – Moneywort

increase(foo[1m]) will do the same as rate(foo[1m]) * 60 (with some magic thrown in). Do note, however, that computing a rate over 1 minute every minute will throw away one data point every minute, so you should at the very least use a step that's lower than 1 minute. – Machicolation

Thanks everyone! Much appreciated! – Mayday

Answer 2

If you need to calculate per-minute increase rate for a counter metric, then use increase(...[1m]). For example, the following query returns the increase of processed_work_items_total{job="MainWorker"} time series over the last minute:

increase(processed_work_items_total{job="MainWorker"}[1m])

Note that the increase() function in Prometheus may return unexpected results due to the following issues:

• It may return fractional results over integer metric because of extrapolation. See this issue for details.
• It may miss counter increase between the last raw sample just before the lookbehind window specified in square brackets and the first raw sample inside the lookbehind window.
• It may miss the initial counter increase at the beginning of the time series.

These issues are going to be addressed in Prometheus according to this design doc. In the mean time it is possible to use MetricsQL, which is free from these issues.

Marcelinomarcell