Airflow sends metrics by statsd but not all of them

Question

Recently I have updated my airflow.cfg to enable metrics thought Statsd.

I have injected this settings to airflow.cfg:

I'm injecting this configuration:

AIRFLOW__SCHEDULER__STATSD_ON=True
AIRFLOW__SCHEDULER__STATSD_HOST=HOSTNAME
AIRFLOW__SCHEDULER__STATSD_PORT=9125
AIRFLOW__SCHEDULER__STATSD_PREFIX=airflow

I'm not using standard Statsd service, but Statsd-exporter which use Statsd protocol, so from my knowledge I can point directly Airflow to send metrics to Stats-exporter. By default it works on 9125 port.

After Statsd-exporter receives metrics, Prometheus is able to scrape them in regular manner.

All fine, all good. Moreover I made my mapping file to Statsd-exporter where I use a bit regex but.... my issue is that when I open WEB UI of Statsd-exported (9102 port) I see part of Airflow metrics, but not all of them!.

Documentation says about list of metrics here

For instance I see that Airflow sends: ti_failures, ti_successes, dagbag_size etc. But there are completely no metrics like: dag...duration or executor.open_slots and couple others.

Really big thank you for anyone who ever played with Statsd and Airflow, as I have no clue:(

Orrin

Answer 1

I recently instrumented airflow and its metrics to be exported from statsd to prometheus. In my architecture I have airflow running in Kubernetes pods and these specifically are:

• scheduler
• worker
• flower
• web

Only scheduler, worker and web have a side-car container to export statsd metrics (let's call these metrics pods).

And the list of metrics you see in official docs (https://airflow.apache.org/metrics.html) are not available to all metric pods.

To adhere to your specific problem, dag...duration are exported by worker node.

Cryptocrystalline

I have edited worker node's configuration file and set statsd_on=True. Still, I'm not getting any metrics of scheduler or airflow on 9102 port. I have used this command for running the statsd exporter. docker run -d -p 9102:9102 -p 9125:9125 -v /tmp/statsd_mapping.yml:/tmp/statsd_mapping.yml prom/statsd-exporter --statsd.mapping-config=/tmp/statsd_mapping.yml – Leesa