How can one use HashiCorp Vault in Airflow?

Question

I am starting to use Apache Airflow and I am wondering how to effectively make it use secrets and passwords stored in Vault. Unfortunately, search does not return meaningful answers beyond a yet-to-be-implemented hook in Airflow project itself.

I can always use Python's hvac module to generically access Vault from PythonOperator but I was wondering if there is any better way or a good practice (e.g. maybe an Airflow plugin I missed).

Confirm

Answer 1

Airflow >=1.10.10 supports Secrets Backends and supports getting Airflow Variables and Connections from Hashicorp Vault.

More Details in Airflow Docs: https://airflow.apache.org/docs/stable/howto/use-alternative-secrets-backend.html#hashicorp-vault-secrets-backend

If you want to test it locally check the tutorial at https://www.astronomer.io/guides/airflow-and-hashicorp-vault/

Set the following config in airflow.cfg, update based on your environment:

backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend
backend_kwargs = {"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "http://127.0.0.1:8200"}

Example DAG to test the integration:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from airflow.hooks.base_hook import BaseHook


def get_secrets(**kwargs):
    conn = BaseHook.get_connection(kwargs['my_conn_id'])
    print(f"Password: {conn.password}, Login: {conn.login}, URI: {conn.get_uri()}, Host: {conn.host}")

with DAG('example_secrets_dags', start_date=datetime(2020, 1, 1), schedule_interval=None) as dag:


    test_task = PythonOperator(
        task_id='test-task',
        python_callable=get_secrets,
        op_kwargs={'my_conn_id': 'smtp_default'},
    )

Emu

Is it possible to renew vault token programmatically from the hook or operator ? – Balance

@Balance Trying to figure this out as well – Jdavie

No, it is currently not possible to renew vault token natively. – Emu

same here. our tokens expire every 12 hours. the closest thing i got regarding renew is this: 1. install the vault command line tool within my airflow docker image 2. set backend_kwargs = {..., "token_path": "/usr/local/airflow/.vault-token", ...} in airflow.cfg 3. ask a leprechaun to go into each airflow container and do $ vault token renew every 6 hours since i'm told by the internet that setting up a cron job in the docker container is difficult and not recommended – Tamis

Does anybody know whether I can actually use a path to the connection? For instance, specify the top hierarchy as connections_path: backend_kwargs = {"connections_path": "kv"} and then when calling the PythonOperator in the example above, use a path as op_kwargs={'my_conn_id': '/my_secrets/dev/smtp_default'} so that the connection inferred from Vault would then be kv/my_secrets/dev/smtp_default? – Pisistratus

No you can't, you will have to set: {"connections_path": "kv/my_secrets/dev/"} – Emu

I would say for all except a very dedicated vault secret mount (which may or may not be available to your company), the integration is severely lacking. It imposes it's own structure, and that (in our case, and I suspect others), made it more or less worthless – Gobbler

airflow cannot read secret values from vault if i set variables_path to other than variables? any idea how i can read values from other paths than variables this doesn't work "variables_path": "my_custom_variables" – Toboggan

Answer 2

Additional things to keep eye on If you are new to Vault:

Validate that your airflow.cfg backend_kwargs parameter mount_point really exists in vault: vault secrets list

Try your DAG code above with command line first before trying it in web interface airflow test example_secrets_dags test-task 2020-06-04

Fallen

Answer 3

How to config with SSL vault backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend backend_kwargs = {"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "https://127.0.0.1:8200"}

Brigham