Airflow metrics with prometheus and grafana

Question

any one knows how to send metrics from airflow to prometheus, I'm not finding much documents about it, I tried the airflow operator metrics on Grafana but it doesnt show any metrics and all it says no data points.

Illsorted

Hi, Priyal Patil have you find the solution? I installed statsd, which listen at one port 9125 towards airflow the other port 9102 for Prometheus. but port 9102/metrics only some cpu/memory related metrics, there is no airflow metrics. I have run "pip install 'apache-airflow[statsd]' within the airflow pod – Notice

Yes I did, I ran the airflow-statsd-exporter container with arguments "--statsd.listen-udp=:9125 --web.listen-address=:9102" and here is the airflow cfg - statsd_port = 9125 – Illsorted

Answer 1

By default, Airflow doesn't have any support for Prometheus metrics. There are two ways I can think of to get metrics in Prometheus.

1. Enable StatsD metrics and then export it to Prometheus using statsd exporter.

2. Install third-party/open-source Prometheus exporter agents (ex. airflow-exporter).

If you are going with 2nd approach then the Airflow Helm Chart also provides support for that.

Edit

If you're using statsd exporter here is a good resource for Grafana Dashboard and exporter config.

Randy

Hey sshah, thank you for your reply, I tried the 1st approach- enabled statsd and installed prometheus exporter and I dont see any airflow stats being sent over there- – Illsorted

I am not very much sure about it as I never worked on this path. – Randy

Answer 2

This is how it worked for me -

• Running airflow in docker using this doc
• Added this configuration inside the docker-compose file downloaded in the previous step AIRFLOW__SCHEDULER__STATSD_ON: 'true'
AIRFLOW__SCHEDULER__STATSD_HOST: statsd-exporter
AIRFLOW__SCHEDULER__STATSD_PORT: 9125
AIRFLOW__SCHEDULER__STATSD_PREFIX: airflow
  Under environment section
• Now run the statsd_export
• docker run -d -p 9102:9102 -p 9125:9125 -p 9125:9125/udp \ -v $PWD/statsd_mapping.yml:/tmp/statsd_mapping.yml \ prom/statsd-exporter --statsd.mapping-config=/tmp/statsd_mapping.yml
• Get the statsd_mapping.yml contents from Here
• Now do docker-compose up to run the airflow and try to run some worflow and you should see logs at http://localhost:9102/metrics

Salvidor

Answer 3

If you installed your Airflow with statsd support:

pip install 'apache-airflow[statsd]'

you can expose Airflow statsd metrics in the scheduler section of your airflow.cfg file, something like this:

[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

Then, you can install a tool called statsd_exporter, that captures statsd-format metrics and converts them to Prometheus-format, making them available at the /metrics endpoint for Prometheus to scrape.

There is a docker image available on DockerHub called astronomerinc/ap-statsd-exporter that already maps Airflow statsd metrics to Prometheus metrics.

References:

• https://airflow.apache.org/docs/stable/metrics.html
• https://github.com/prometheus/statsd_exporter
• https://hub.docker.com/r/astronomerinc/ap-statsd-exporter/tags

Benjamin

Hey @holypriest, I have tried that but it doesnt seem to send the stats to ap-statsd-exporter – Illsorted

That's odd. Can you confirm that the containers are in the same network and talking with each other? I use the same approach here and it works like a charm. Try debugging it executing bash inside the containers and executing some commands to test the healthiness of the services and such. – Benjamin

If I install statsd_exporter where airflow is running, does mapping happen automatically. – Illsorted

Not exactly, it depends on your infrastructure. Airflow exposes statsd metrics on port 8125 of the host, statsd_exporter shoud be somehow configured to watch that port and expose the data to prometheus on another port, and then prometheus should be configured to watch the statsd_exporter port. I used this setup on a Kubernetes cluster, so I made sure to expose/watch the right ports for each one. You should also check if all the services are responding. – Benjamin