How grafana dashboard filter Zipkin (tracing, Explore) metrics?

Question

I'd like to ask about zipkin with grafana dashboard. I installed zipkin with grafana using official docker-compose (but storage is elasticsearch. because when I use mysql, an access error happens. es doesn't happen) https://github.com/openzipkin-attic/docker-zipkin#elasticsearch

By the way, I am using zipkin for collecting graphQL execution time with nodejs application. (using opentelemetry library such as HttpInstrumentation, ExpressInstrumentation and GraphQLInstrumentation) And I'd like to show which graphQL queries are slow on some graphical chart.

I try to use prometheus metrics explorers below with the grafana dashboard.

zipkin_collector_messages_total
zipkin_collector_bytes_total
zipkin_collector_spans_total
zipkin_collector_spans_dropped_total
zipkin_collector_messages_dropped_total

But it looks like they can't filter any metrics tags on grafana dashboard ... (because in terms of above controller, we can only use 'instance, job, transport' labels)

Could we do that by prometheus metrics explorers? Or, for filtering by metrics, should we use data storage (such as elastic search, mysql, etc) on grafana dashboard? Thank you so much.

Foreknowledge

my backend application is made by node.js – Foreknowledge

Answer 1

You need to make distinction between traces and metrics. Your saved metrics in the Prometheus are only aggregated data from traces - and it looks like they don't have required dimension for you.

Configure your OTEL collector properly (use spanmetricsprocessor processor - that still unstable processor! - customize dimension config for your need - e.g. graphl.query - sorry I really don't know your tags, so it is blind guess) and export metric to the prometheus. Then you will have in the prometheus also metrics with usefull dimension for your use case and you can filter/group on that.

But I would simply use Zipkin UI directly and filter by min duration (+ tags eventually) to find traces/queries with the longest duration/latency.

Sectarianism

This is a quick note to thank you for giving me very polite detail. I'll take a look and exploring. – Foreknowledge

I'm seeking how we implement this. Excuse me, could this processor be used by not only Golang project? I couldn't find out another language solution. (Currently we use node.js) – Foreknowledge

@Foreknowledge of course, collector can be standalone application and nodejs instrumentation sends traces to that collector, which calculates metrics from the traces – Sectarianism

Jan Garaj, thank you for answering. Oh I see. I tried to implement it but it's difficult for me in this time by node.js (It looks like there is no example) .... I think it's relatively easy to use Kibana dashboard on elasticsearch with zipkin rather than Grafana for viewing ranking. I switch to technology stack. – Foreknowledge

Tracing in the Grafana can be a challenge. IMHO Grafana AWS X-Ray plugins is the most mature atm, because it provides also Trace Statistics mode. – Sectarianism