How Grafana resolution works when showing more points than Prometheus have

Question

I scrape metrics with Prometheus every 30 seconds. When I check in Prometheus graph:

elasticsearch_jvm_memory_max_bytes{}[2m]

I can see there is range vector with 4 values:

2095185920 @1626523484.001
2095185920 @1626523514.001
2095185920 @1626523544.001
2095185920 @1626523574.001

That makes sense, 30s * 4 values = 2m. When I run the same command in Grafana, I can see data points every 15s:

In the example above, there is 3 data points in 14:43:30, 14:43:45 and 14:44:00. I can see this with query resolution 1/1. If I set resolution to 1/2, graph looks normal, with data points every 30s. I read about resolution in grafana and maybe I did not understat correctly, but I would expect that 1/1 should be one point in panel per 1 data point from prometheus. 1/2 should show 1 point in panel per 2 data points from prometheus, etc... Could anybody explain me, what am I missing or why grafana works like this (with some example)?

Grafana: v7.3.6

Thank you.

Tamah

To understand your question I tried jvm_memory_max_bytes{} which is working but jvm_memory_max_bytes{}[2m] gives me an error: "invalid expression type \"range vector\" for range query, must be Scalar or instant Vector". – Lundeen

Answer 1

Grafana sends request to /api/v1/query_range endpoint at Prometheus when it needs points for building the graph. Grafana sends the following args to the endpoint:

• query - the PromQL query to execute
• start and end - the time range for the graph
• step - the interval between points on the graph

This endpoint always returns points with timestamps start, start+step, start+2*step, ..., start+N*step, where start+N*step <= end. Prometheus executes the provided query independently per each such timestamp. Obviously, the real samples stored in Prometheus can be missing at the requested timestamps. That's why Prometheus returns sample values with the closest timestamps. See these docs for details.

Pisgah

Your explanation makes sense, but what is a good work around? I just want 1/1 resolution in my grafana graphs. – Pocketknife

Unfortunately Grafana datasource for Prometheus cannot query and display raw samples stored in Prometheus. You can query raw samples and process them on your side if this is needed - valyala.medium.com/… . – Pisgah

Turns out I was using $__interval when I should have been using $__rate_interval for a rate query – Pocketknife