Prometheus / Grafana highest value and time

Question

Is it possible in grafana with a prometheus backend to determine the highest value recorded for the lifetime of a data set, and if so, determine the time that the value occurred?

For example, I'm using site_logged_in as the query in a Singlestat panel to get the current number of logged in users, along with a nice graph of recent activity over the past hour. Wrapping that in a max() seems to do nothing, and a max_over_time(site_logged_in[1y]) gives me a far too low number.

The value is a single gauge value coming from the endpoint like so

# HELP site_logged_in Logged In Members
# TYPE site_logged_in gauge
site_logged_in 583

Is something like determining highest values even a realistic use case for prometheus?

Caster

Answer 1

max_over_time(site_logged_in[1y]) is the max over the past year, however this presumes that you have a year worth of data to work from.

Stiletto

Does that conflict at all with grafana's resolution / min step / time range features? I've had spikes over 500, but max_over_time is returning about 460. We implemented prometheus 4 months ago, would setting the range higher than the available data throw it off? – Caster

Those only affect graphs, and would make little difference for a range this long anyway. Most likely your retention period is less than a year, so you're only have more recent data. – Stiletto

Yup, looks like I screwed something up in the config. It was keeping 15 days. It's now set to 150 days, close enough for what we're going for. Thanks for your help! – Caster

Answer 2

The highest value over the specified time range can be obtained with max_over_time() function. For example, the following value would return the maximum value for site_logged_in metric over the last year:

max_over_time(site_logged_in[1y])

Unfortunately Prometheus doesn't provide the function for returning the timestamp for the maximum value. If you need to obtain the timestamp for the maximum value, then you can use tmax_over_time() function from MetricsQL. For example, the following MetricsQL query returns the timestamp in seconds for the maximum value of site_logged_in metric over the last year:

tmax_over_time(site_logged_in[1y])

Fairly