Stacked Chart from Prometheus data in Grafana

Question

I've been trying to create a stacked bar chart in Grafana from Prometheus data but struggling and to be honest, I'm not even entirely sure that what I'm trying to do is possible. I've had a go using the old graph panel and the new bar chart panel - but to be honest I'm finding the new bar chart panel fairly unintuitive and documentation is vague. I'm working with a time-series counter for nginx requests split by response code. A sample of the data in Prometheus is:

nginx_vts_server_requests_total{code="2xx", host="example.com", instance="localhost:443", job="nginx_vts"}

I was hoping to create a stacked chart based on the number of requests from the last 12 hours, taking the last value of:

increase(nginx_vts_server_requests_total[12h])

I haven't included an aggregate as I'm only working with a single instance.

I am hoping to create something similar to the this image (mocked up in excel):

If anyone has created such a chart before in Grafana based on Prometheus data, I'd be very grateful if you could show me how it's done (if it is in fact possible at all). The screenshot at the top of the page at https://grafana.com/docs/grafana/latest/visualizations/bar-chart/ suggests that it is possible to do such groupings although this example is not stacked.

Hemialgia

Answer 1

I struggled with a similar problem. This is the query I used; which is quite similar:

sum(rate(http_requests_received_total[1h])) by (code,instance)

• First make sure the Bar Chart option Stacking is set to Normal.
• Set the query Type to be Instant since we'll only use the last value.
• Go on to Transform the query. Remember the Table view option is your friend in grasphing what is going on. Add in this order:
  • Add Reduce transform to reduce Series to rows. I use Max-calculations, but remember we really have just one.
  • Add Extract fields transform to split Field into separate fields.
  • If you want to do anything with the ordering this is the place to add a Sort by transform.
  • Add Grouping to matrix transform to create new columns from the new code field. Use code as Column, instance as Row and Max as Cell Value.

PS: If you get errors use Grafana Explore to check that the values look the way you imagine them. Be especially on the lookout for empty values on codeand instance.

Banian

Answer 2

This is for future reference purposes, the key part from @thoredge response that worked for my case is the Grouping to matrix transform.

The query I had in prometheus was:

sum (gitlab_ci_pipeline_job_duration_seconds{job_name=~"Random job matching.*"}) by (ref,job_name)

The data transformation I configured was:

The results look like:

Sillimanite