Simple cumulative increase in Prometheus

Question

I have an application that increments a Prometheus counter when it receives a particular HTTP request. The application runs in Kubernetes, has multiple instances and redeploys multiple times a day. Using the query http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"} produces a graph displaying cumulative request counts per instance as is expected.

I would like to create a Grafana graph that shows the cumulative frequency of requests received over the last 7 days.

My first thought was use increase(...[7d]) in order to account for any metrics starting outside of the 7 day window (like in the image shown) and then sum those values.

I've come to the realisation that sum(increase(http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"}[7d])) does in fact give the correct answer for points in time. However, resulting graph isn't quite what was asked for because the component increase(...) values increase/decrease along the week.

How would I go about creating a graph that shows the cumulative sum of the increase in these metrics over the passed 7 days? For example, given the simplified following data

| Day | # Requests |
|-----|------------|
| 1   | 10         |
| 2   | 5          |
| 3   | 15         |
| 4   | 10         |
| 5   | 20         |
| 6   | 5          |
| 7   | 5          |
| 8   | 10         |

If I was to view a graph of day 2 to day 8 I would like the graph to render a line as follows,

| Day | Cumulative Requests |
|-----|---------------------|
| d0  | 0                   |
| d1  | 5                   |
| d2  | 20                  |
| d3  | 30                  |
| d4  | 50                  |
| d5  | 55                  |
| d6  | 60                  |
| d7  | 70                  |

Where d0 represents the initial value in the graph

Thanks

Xylotomous

Did you ever figure out a way to do this? I am attempting to do the same and coming up empty – Hideout

I'm afraid we didn't – Xylotomous

Is there still no way to do this? – Arronarrondissement

Answer 1

If I understood your question's idea correctly, I think I managed to create such graph with a query like this

sum(max_over_time(counterName{someLabel="desiredlabelValue"}[7d]))

A graph produced by it looks like the blue one:

The reasons why the future part of the graph decreases are both because the future processing hasn't obviously yet happened and because the more-than-7-days-old processing slides out of the moving 7-day inspection window.

Balkhash

Answer 2

Prometheus doesn't provide functionality, which can be used for returning cumulative increase over multiple time series on the selected time range.

If you still need this functionality, then try VictoriaMetrics - Prometheus-like monitoring solution I work on. It allows calculating cumulative increase over multiple counters. For example, the following MetricsQL query returns cumulative increase over all the time series with http_requests_total name on the selected time range in Grafana:

running_sum(sum(increase(http_requests_total)))

How does it work?

1. It calculates increase per each time series with the http_requests_total name. Note that the increase() in the query above doesn't contain lookbehind window in square brackets. VictoriaMetrics automatically sets the lookbehind window to the step value, which is passed by Grafana to /api/v1/query_range endpoint. The step value is the interval between points on the graph.

2. It sums increases returned at step 1 with the sum() function individually per each point on the graph.

3. It calculates cumulative increase over per-step increases returned at step 2 with the running_sum function.

Bucksaw