Grafana graphing HTTP requests per minute with http_server_requests_seconds_count

Question

I have a Spring boot app throwing out open metric stats using micrometer.

For each of my HTTP endpoints, I can see the following metric which I believe tracks the number of requests for the given endpoint:

http_server_requests_seconds_count

My question is how do I use this in a Grafana query to present the number number of requests calling my endpoint say every minute?

I tried

http_client_requests_seconds_count{}

and

sum(rate(http_client_requests_seconds_count{}[1m]))

but neither work.

Thanks in advance.

Dump

Answer 1

rate(http_client_requests_seconds_count{}[1m]) will provide you the number of request your service received at a per-second rate.

However by using [1m] it will only look at the last minute to calculate that number, and requires that you collect samples at a rate quicker than a minute. Meaning, you need to have collected 2 scrapes in that timeframe.

increase(http_client_requests_seconds_count{}[1m]) would return how much that count increased in that timeframe, which is probably what you would want, though you still need to have 2 data points in that window to get a result.

Other way you could accomplish your result:

increase(http_client_requests_seconds_count{}[2m]) / 2 By looking over 2 minutes then dividing it, you will have more data and it will flatten spikes, so you'll get a smoother chart.

rate(http_client_requests_seconds_count{}[1m]) * 60 By multiplying the rate by 60 you can change the per-second rate to a per-minute value.

Here is a writeup you can dig into to learn more about how they are calculated and why increases might not exactly align with integer values: https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates

Turenne

Answer 2

The http_client_requests_seconds_count metric is a counter. This means it may increase over time and may reset to zero when the service, which exposes this metric, is restarted. This metric counts the number of requests since the last service restart according to its name (see naming conventions for Prometheus metrics).

Prometheus provides increase() function for calculating the counter increase on the lookbehind window specified in square brackets (see time duration format for possible values, which can be specified in square brackets). So, increase(http_client_requests_seconds_count[1m]) returns the number of requests during the last minute aka requests per minute.

Unfortunately, increase() function in Prometheus has the following issues, which may lead to unexpected results:

• It needs at least two raw samples on the specified lookbehind window. Otherwise it returns empty response.
• It may return fractional results when applied to integer counters because of extrapolation. See this issue for details.
• It may return lower than expected results, since it misses the increase between the first raw sample in the lookbehind window and the previous raw sample.

Prometheus developers are going to fix these issues - see this design doc. In the mean time VictoriaMetrics can be used - its' increase() function is free of issues mentioned above.

Fibrinolysin