extract single label as column in log panel

Question

I've just started exploring Loki to aggregate my logs and I'm assuming I've missed something obvious.

I have a bunch of machines/vms/containers with promtail forwarding systemd's journal to a single loki server (just the default example config) and I can explore them just fine. I can filter the logs by any of the labels as required.

However, on a dashboard panel I have tried both the "table" and "logs" panel types and can't seem to do what seems extremely basic to me. I want filter by some of my labels and then extract one or more of the remaining labels as a column to be displayed. The closest I have found is to turn on "Unique labels" in the "logs" panel type which just throws all of the values in one column together.

What have I missed?

Skijoring

Answer 1

You can use the table panel type. Then go to the Transform tab which is located next to the Query tab at the bottom. Add an Extract fields transformation and for the Source dropdown select Labels. Then press Add path button and enter the name of the label you want as a column.

Note that there is also an Organize Fields transform which is useful to control the visibility and ordering of columns.

Nolanolan

Thanks very much - this seemed to be exactly what I was after. – Skijoring

This only works if the logs are JSON-formatted or have key-value pairs. – Altazimuth

Answer 2

Just starting out also, so please don't take this as authoritative.

I made some progress by:

1. processing the log "Line" using a regexp to create capture groups as referable labels, and

2. using the line_format to re-assemble the log line in the desired format

   ... | regexp ^(?P<thread><.*>)\s+(?P<channel>\[.*\])\s+(?P<message>.*)$ | line_format "{{.env}} {{.node}} {{.thread}} {{.channel}} {{.message}}"

Hope it helps.

Update:

You can also prepend information using the template stage when sending, something like as follows (in the agent config):

- template:                  
    source: message                  
    template: '{{ .node }} {{ .message }}'

Kimberlykimberlyn

Thanks for the suggestion - that's mangling the log-line itself though, which wasn't what I was after. I suppose I could keep the original line after the newly regexp-extracted repetition of the label. – Skijoring

You can also mangle it at the sender (updated answer), but I agree that both feel a bit hacky – Kimberlykimberlyn

While I can get labels/fields I want in the data by configuring promtail with a regex, I'm still confused about the (non-)possibilities to have columns in a logs panel, i.e. some table functionality in a logs panel. It seems, surprisingly, that there is no solution for that yet? (github.com/grafana/grafana/discussions/42315) – Maxey

Answer 3

Use line_format and the line function in the following form:

... | line_format "{{(printf \"%-7s\" .node)}} {{__line__}}"

assuming that node is the label, and you want those printf format specifiers (adjust to taste).

line_format allows any go template argument, so should be able to achieve the need I think.

Note: for performance reasons it is preferred to put such processing after any filtering steps.

Kimberlykimberlyn