PromQL Requests per minute

Question

I'm trying to create a graph of total POST requests per minute in a graph, but there's this "ramp up" pattern that leads me to believe that I'm not getting the actual total of requests per minute, but getting an accumulative value.

Here is my query:

sum_over_time(django_http_responses_total_by_status_view_method_total{job="django-prod-app", method="POST", view="twitch_webhooks"}[1m])

Here are the "ramp up" patterns over 7days (drop offs indicating a reboot):

What leads me to believe my understanding of sum_over_time() is incorrect is because the existing webhooks should always exist. At the time of the most recent reboot, we have 72k webhook subscriptions, so it doesn't make sense for the value to climb over time, it would make more sense to see a large spike at the start for catching webhooks that were not captured during downtime.

Is this query correct for what I'm trying to achieve?

I am using django-prometheus for exporting.

Lorsung

Answer 1

You want increase rather than sum_over_time, as this is a counter.

Beastly

I did think so as well, thanks for confirming. An issue I see, though, is this same ramp up trend, and if I look over the last 24 hours it accumulates all the way as far as 100k+ per minute which is definitely inaccurate. I simply used the same query as above using increase() rather than sum_over_time(), is that correct? – Lorsung

Accepting answer. Appears my actual issue is that there are multiple instances of Prometheus spawning somehow which cause these fluctuations when being treated as a single target. – Lorsung

Answer 2

If the django_http_responses_total_by_status_view_method_total metrics is a counter, then increase() function must be used for returning the number of requests during the last minute:

increase(django_http_responses_total_by_status_view_method_total[1m])

Note that increase() function in Prometheus can return fractional results even if django_http_responses_total_by_status_view_method_total metric contains only integer values. This is due to implementation details - see this comment and this article for details.

If the django_http_responses_total_by_status_view_method_total metric is a gauge, which shows the number of requests since the previous sample, then sum_over_time() function must be used for returning requests per last minute:

sum_over_time(django_http_responses_total_by_status_view_method_total[1m])

Pariah