Get Total requests in a period of time

Question

I need to show, in Grafana, a panel with the number of requests in the period of time selected in the upper right corner.

For this I need to solve 2 issues here, I will ask the prometheus question here and the Grafana question in another link.

If I have a Counter http_requests_total, How can I build a query to get an integer with the total number of requests during a period of time (for example:24hs)?

Hospitalet

Note that Prometheus may return fractional value from increase() function on a time series with integer values. It may also miss some values on slowly increasing time series. Both issues are documented at github.com/prometheus/prometheus/issues/3746 . If you need accurate integer values from increase() function, then take a look at MetricsQL. – Mcginnis

Answer 1

What you need is the increase() function, that will calculate the difference between the counter values at the start and at the end of the specified time interval. It also correctly handles counter resets during that time period (if any).

increase(http_requests_total[24h])

If you have multiple counters http_requests_total (e.g. from multiple instances) and you need to get the cumulative count of requests, use the sum() operator:

sum(increase(http_requests_total[24h]))

See also my answer to that part of the question about using Grafana's time range selection in queries.

Conqueror

What if period is not last 24h, but from fist date&time and second date&time? – Firmament

@Cherry, You can use an offset, e.g. increase(http_requests_total[5h] offset 1d) or increase(http_requests_total[357s] offset 123m). This way you specify the width of the period you interested in and how far in the past this period is. But this is definitely NOT a convenient way. Maybe somebody else can suggest a more practical solution to your question. @donotreply's answer looks like what you ask for, but seems to be applicable only when using Graphana. – Conqueror

Actually on server restart prometheus values get reset to 0, so the graph suddenly drops, if we see the increase of 24 hr, it comes inaccurate as it is the difference of the first and last value , any better approach to this ? – Stockinet

@somyabhargava I had the exact problem - I found the answer on #55928579 . So in this case it would be sum(increase(http_requests_total[100y])) – Aboutface

But sum(increase(http_requests_total[100y])) will get you the total value over the whole lifetime of the counter and not just the selected time interval – Yb

Yes exactly what @JellyFilledNuts said, this will be a whole lifetime counter – Stockinet

What about count of bucket elements? If I run job_timer_myjob_bucket{le="+Inf"} it returns number of items in the bucket e.g. 100. Now I would like to get the same count for specified interval e.g. for the previous day. If I use increase(job_timer_myjob_bucket{le="+Inf"}[1h]) it return me not integer - e.g. 55.8 is returned. – Wan

Answer 2

SO won't let me comment on Yoory's answer so I have to make a new one...

In Grafana 5.3, they introduced $__range for Prometheus that's easier to use:

sum(rate(http_requests_total[$__range]))

This variable represents the range for the current dashboard. It is calculated by to - from

http://docs.grafana.org/features/datasources/prometheus/

Mordred

note: for me, $__range does work, but grafana does not show it as an available variable in its helper tooltip/popup. go hardcore and type it out. – Tun

Answer 3

As per increase() documentation, it is not aggregation operator. Thus, it will give wrong answer. (See note.)

You should use sum_over_time() function which aggregates over time interval.

sum_over_time(http_requests_total[24h])

If you have multiple counters, use sum() operator:

sum(sum_over_time(http_requests_total[24h]))

Note: I have 5 datapoints which has values: 847, 870, 836, 802, 836. (updated every minute)

increase(http_requests_total[5m]) returns 2118.75 

sum_over_time(http_requests_total[5m]) returns 4191

Contrived

sum_over_time shouldn't be used directly on counters, use after rate. – Ribble

When I use sum(sum_over_time(http_requests_total[$__interval])) I am still seeing drops in the graph while instead it should be monotonously increasing. – Skillern

The data points are not monotonically increasing. Is the counter resetting every minute and you're collecting every minute? Are you performing a push method of getting data into prometheus? This would explain why sum_over_time works for you. For others who's data points would be scraped a reset cannot happen every minute, so you would find counters that look like this: 847, 1690, 2412, 3245, 4023. In this case the increase function should work better. – Acquire

Answer 4

http_requests_total - http_requests_total offset $__interval > 0

This builds off another answer and comment that works and handles restart situations.

The offset keeps the value always as an integer and does not try to perform interpolation like the increase and rate functions.

The > 0 filter at the end will ignore all of the negative values that could be captured due to a restart.

The end result is the accurate total number of requests over time if you choose to chose the total value in the legend.

Forsook

All the answers using increase() didn't work across all dashboard ranges, this one does! – Pearse

Answer 5

Solution: In order to calculate sum of https counters on prometheus grafana you should use increase method and set generic Time Range $interval in order to sum and calculate all http requests counters.

increase(http_requests_total[$interval])

According to Prometheus Reference:

increase() increase(v range-vector) calculates the increase in the time series in the range vector. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for. The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments.

The following example expression returns the number of HTTP requests as measured over the last 5 minutes, per time series in the range vector:

increase(http_requests_total{job="api-server"}[5m]) increase should only be used with counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.

P.S

1. You should set the correct Quick range on Grafana for setting the right time frame you choose (that straight rendered to $interval variable) In addition I suggest to set on the Graph visualisation the right resolution and Min time interval ( in your case it's per day -> 1d)

2.In order to sum all amount of requests just perform sum function

sum(increase(http_requests_total[$interval]))

Sphygmograph

I used sum(increase(http_requests_total[$interval])) query and visualized it in Grafana bar chart. I would expect that the far right bar would display today's data. However, it displays yesterday's data. Moreover, the date below the bar is incorrect, it is one day in the future (it displays 7-Apr-2022 although yesterday was 6-Apr-2022). – Greyhound

Answer 6

To get the exact count for the last 24 for hours I have created the following query:

max_over_time(http_requests_total[6s])- min_over_time(http_requests_total[24h])

Note: works for me :)

Simarouba

This will give you wrong results if http_requests_total has been reset during those 24 hours. – Yb

Answer 7

The easiest way to do this in the modern version of Grafana is to set the Calculation field of the Value options section to "Difference" reduce function:

The query is your http_requests_total data without any aggregation.

Cirri

thx, this is the most accurate answers – Glengarry

..but this aproach won't be restart proof, that's the disadvantage – Glengarry

Answer 8

It seems to me that all the previous answers have misinterpreted the questions, which is to get a count from t0 to t1, where the value at t0 should be 0.

For this one can use the @ modifier as per the documentation https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier:

http_requests_total - http_requests_total @ start()

Amado

I think this one will not give the correct result in case of counter reset. – Morpho

@DuyTran why would you make a counter reset? isn't the counter should always start counting from 0 everytime you pick t0 and t1 ? – Dominquedominquez

@YosraMH it resets in case of your app restarted or u set the counter to 0 in purpose. Normally, t0->0, t10->10 then above func works. In case of counter reset at t10, t10->0. The increase is now 0. right? – Morpho

Answer 9

sum(increase(http_requests_total[$__range])) worked for me. $__range is grafana variable which reflects time range selected in top right of grafana.

It is highly recommended to use increase to automatically adjust counter for resets.

Increase results in fractional values due to extrapollation at boundaries as well. Rounding functions with the query also worked for me.

round(sum(increase(http_requests_total[$__range])))

or

Since you are using grafana you can set decimals to 0 in standard options. (in grafana 9.1.8 this is not working, its bug). So I have set this in panel json as below.

  "fieldConfig": {
    "defaults": {
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "value": null,
            "color": "green"
          },
          {
            "value": 80,
            "color": "red"
          }
        ]
      },
      "color": {
        "mode": "thresholds"
      },
      "decimals": 0,
      "unit": "none"
    },
    "overrides": []
  }

Note: These queries are tried with grafana stat panel. (grafana version 9.1.8)

Deckard

Answer 10

Recently, I have the confusion too, and I got some solutions for it, but all of them are not all work perfectly.

---

solution 1:

    sum(increase(your_point))[$__interval]

This one will cause some different value with same statement, and also will cause the zero-value (actually not zero).

---

Solution 2:

    max_over_time(your_point[$__range])- min_over_time(your_point[$__range])

    your_point[$__range] -  your_point offset $__range

Both of this have potentially bug (value reset), and only can get the value in [sometime-now], cannot get the answer in anytime period.

---

Solution 3:

    sum_over_time(your_point)[$__range]

This solution will take much time to change your metrics (reset in certain period), but truely work.

Is anyone can give me another solution?

Brocade

Solution 3 doesn't work for me: " parse error: ranges only allowed for vector selectors: details:" – Hoggard

I guess that is typo in solution 3, range vector selector cannot be applied to result of a query (derived value). Range vector selector always applied to raw values (timeseries stored in TSDB) hence it is sum_over_time(your_point[$__range]). However Iam not supporting this solution. – Deckard

Answer 11

To get the accurate total requests in a period of time, we can use offset:

http_requests_total - http_requests_total offset 24h

increase will extrapolate the range so that we can see float number in the result.

By using offset, the value is always integer because it just calculates the difference between start and end

Excellent

Thanks a lot! This is a most accurate request to gain real results from *_total countes. A bit more universal approach for fine-grained intervals is "http_requests_total - http_requests_total offset $__interval" (for grafana) – Gwenngwenneth

No, this is not the correct solution. If the instances are restarted, the counter will be reset. So .... – Monosaccharide