Grafana state timeline panel with values (states) supplied by label

Question

I do have a Prometheus time series with samples like these:

a_metric{band="1", state="A"} 1
a_metric{band="2", state="C"} 1
a_metric{band="1", state="A"} 1
a_metric{band="2", state="C"} 1
a_metric{band="1", state="B"} 1
a_metric{band="1", state="B"} 1
...

I would like to visualize this time series in a state timeline panel such that bands become horizontal bands and states become discrete states within these bands. For this, I would have to extract values from the label state (and use them instead of values 1).

Is this possible and can such a visualization be achieved?

If I understand correctly Prometheus' label_values() cannot serve here, because it is restricted to templating. I suspect Grafana transformations could play a role, but I do not yet have experience with those. The complication also arises because Prometheus does not have string type metrics.

UPDATE Here is a basic image, as requested by @JanGaray.

Fabled

I would add basic image to the question, because I'm not sure what do you want to achieve. – Sejm

The link to state_timeline shows me empty page. What works for me is grafana.com/docs/grafana/next/visualizations/state-timeline – Thurnau

@JanVlcinsky Weird. The link in the post redirects to your URL when I open it. – Potto

Answer 1

suspect Grafana transformations could play a role

It is indeed possible with transformations.

In short

1. Set query format to table
2. Add transformation Grouping to matrix.
   • Set column to band
   • Set row to time
   • Set cell_value to state
3. Add transformation Convert field type
   • Set field to Time/<column>
   • As Time
4. Configure Value mappings for each of the states you have

Explanation

I'll show how the transformations work To see what is happening in the transformations, it is always helpful to enable the as this will show actual returned data to the visualization. Also toggling the transformations on and off using the next to transformation really helps in seeing what is happening.

The state timeline visualization works best with a table as data source. Although it does support time series, the information we want to show is not a time series.

The visualization works best when the data is as follows.

🕓 Time	🇦 Band 1	🇦 Band 2
T1	State A	State B
T2	State C	State A

But when data is formatted as a time series the data will look like this:

🕓 Time	🪟 {Band 1, State A}	🪟 {Band 1, State C}	🪟 ...
T1	1
T2		1

---

So first we need to set query format to from Timeseries to Table. Expend the query options, and set format to Table.

When doing that our table will look like this.

🕓 Time	🇦 band	🇦 state	🪟 Value
T1	Band 1	State A	1
T1	Band 2	State B	1
T2	Band 1	State C	1
T2	Band 1	State A	1

---

Next step is to pivot the table. For this we add the Grouping to matrix transformation.

• Set column to band
• Set row to time
• Set cell_value to state

🇦 Time	🇦 Band 1	🇦 Band 2
1000	State A	State B
2000	State C	State A

---

Now there is just one problem left. The first column should be of data type time but it is of type string. You can see this by the icon infront of the column name. Indicated with emoji in my examples.

But this is an easy fix. Just add Convert field type transformation below Grouping to matrix transformation.

• Set field to Time/<column>
• As Time

Now the table should look as follows:

🕓 Time	🇦 Band 1	🇦 Band 2
T1	State A	State B
T2	State C	State A

Example

Following shows an example for ArgoCD. It shows the sync status for each application over time.

The prometheus exported data from ArgoCD looks as follows:

argocd_app_info{name="Application A", sync_state="Synced", ...} 1
argocd_app_info{name="Application B", sync_state="OufOfSync", ...} 1
argocd_app_info{name="Application C", sync_state="Unknown", ...} 1
argocd_app_info{name="Application D", sync_state="Synced", ...} 1

Norbertonorbie

Note: The Transformation Grouping to matrix is available in Grafana 8.5 or higher. – Misogyny

As a follow-up, in the event of spotty data, you might need to additionally apply a "Sort by" transformation on your Time/<column> field. – Buffon

@Norbertonorbie Do you know if it is still possible to show additional information in a tooltip? E.g. in the above example the column value should be shown in the tooltip. I can not think of a solution because that column is removed by the Grouping to matrix transformation. – Belonging

Answer 2

I don't think it is possible since as you said prometheus is not designed to use strings as values. If you have control over the exporter that is giving you these metrics, you can change the states to different values without the state label, and then use Value Mappings to map: 1 == State A; 2 == State B; etc

Surfacing it this way also will remove the issue of a given band being in multiple states at a given time. Storing the states as labels means your app needs to turn off a given label/state in addition to turning on the next state. If you forget to do this, then the metric will show the band in 2 states at once.

For the blackbox exporter, we get metrics like below with the query probe_success. This can only be 1/0 but the example would work if you have more states as well.

probe_success{instance="https://app1.example.com", job="blackbox"}  0
probe_success{instance="https://app2.example.com", job="blackbox"}  0
probe_success{instance="https://app3.example.com", job="blackbox"}  1
probe_success{instance="https://app4.example.com", job="blackbox"}  1

Set the Value Mappings to map from the number to string

Rebec

The prometheus training resource suggests using labels to report string metrics. I feel like there must be some way to use such metrics? @brian-brazil am I misunderstanding your instruction in the linked page? – Potto

The trouble is that in my case there is an open set of states, so coming up with a mapping up-front is impossible. I know that Prometheus is not designed to use strings as values, but I though perhaps Grafana could use a designated label (its value being a string) instead of actual values from Prometheus (which are constant at 1.0 in this case). This could allow it to accomplish with the Prometheus datasource what is possible with other ones. Does not seem to be the case, though. – Fabled

Answer 3

It is possible to do with https://grafana.com/grafana/plugins/flant-statusmap-panel/

It has exactly same case in the description, and I used it to show statuses of the consul services exported with the consul exporter where the status goes to the status label.

Labrie