How can I measure each API response duration using `http_server_requests_seconds_sum`

Question

In Grafana I want to plot in how long my APIs are taking to return response.

So what I did was create a dashboard, then panel, and there I added these prometheus queries:

• http_server_requests_seconds_sum{uri="/doc"}
• http_server_requests_seconds_sum{uri="/list-doc"}
• http_server_requests_seconds_sum{uri="/cont"}
• http_server_requests_seconds_sum{uri="/list-cont"}
• ... // yes I need to work on API naming as well

Then it gave me a beautiful graph, like this:

I think this graph is actually plotting the summation of the duration, but I want to plot individual values. I can do the opposite, using reduce transformation. But I'm stuck here, your help will be really appreciated.

Detailed explaination:

Let's say I''ve made 4 requests (for the same endpoint)

• request 1 takes 1 sec,
• request 2 takes 2 sec,
• request 3 takes 1 sec,
• request 4 takes 4 sec,

So value http_server_requests_seconds_sum for this endpoint will become:

• 1
• 3 (1+2)
• 4 (3+1)
• 8 (4+4)

What I'm plotting right now is 1,3,4,8

What I want to plot is 1,2,1,4

How can I do so using this http_server_requests_seconds_sum??

I don't want to show average, or rate, I want to render specific duration of each API hit.

Progressist

How do you imagine this should work, if multiple request happen between scrapes? – Confuse

@Confuse I actually don't know... I'm new to grafana and I've mostly found sof a place to learn new tech by questioning... That's why I'm asking... – Progressist

@Confuse I have done it... and I have posted my solution as answer, please check and let me know if there is some modifications you want to add there... thanks... – Progressist

Answer 1

First we have to create a new Gauge and register it, like this:

    private final CollectorRegistry registry = CollectorRegistry.defaultRegistry;

    private final Gauge dictOfMetrics = Gauge.build()
            .name("duration_metric")
            .labelNames("uri")
            .help("duration_metric_help")
            .register(registry);

And then we can use a @Component class implementing HandlerInterceptor, and it has to steps:

• when the request lands on our server, we will add a new header stating the current time
• when the request is serrved, we will get that header, measure time difference, set it to gauge, and remove that header

Can be done like this:

@Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        request.setAttribute("x-start-time", System.currentTimeMillis());
        return true;
    }

@Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
        Long startTime = (Long) request.getAttribute("x-start-time");
        long executeTime = System.currentTimeMillis() - startTime;
        String uri = request.getRequestURI();
        request.removeAttribute("x-start-time");

        dictOfMetrics.labels(uri).set(Double.parseDouble(durationMapString));
    }

Progressist

1. This code is missing definition of durationMapString. 2. This approach in general will give only the last request duration. If there were multiple requests between scrapes, only duration of last one will be exposed, and all the other will be disregarded. – Confuse

Answer 2

Prometheus doesn't have data on every request individually. It scrapes metrics that contain already aggregated data. And as a result it can't show information about individual requests.

Closest to what you described could be achieved with this query:

rate(http_server_requests_seconds_sum[$__rate_interval]) / rate(http_server_requests_seconds_count[$__rate_interval])

It calculates relation of growth of total requests duration to growth of requests count. This value can be reasonable estimation of how long it takes to response to requests.

Confuse