Airflow: Trigger DAG via UI with Parameters/Config

Question

I see that one can trigger_dag with parameters/config key-value pairs using the airflow command line:

For Apache Airflow, How can I pass the parameters when manually trigger DAG via CLI?

In my case, I would like to centralize all operations for airflow via the airflow UI (preferably no CLI access should be granted), which has a button for triggering the dags:

How can I pass the external parameters/configs via the UI?

Gluttony

Answer 1

You're in luck, assuming you're on a recent version of Airflow or can upgrade. Support for triggering a DAG run with a config blob was added in Airflow 1.10.8 under AIRFLOW-5843 in #5843. It's not through the same Trigger DAG icon you've pointed to, but it's through creating a DAG Run from Browse->DAG Runs->Create.

Screenshot of the new form that supports conf copied below from the pull request that added it.

The form will also validate the JSON of the conf which may actually be a step beyond the CLI.

Jecho

Thanks! I'm actually using cloud composer in GCP, and currently the latest version is 1.10.6. I guess we'll have to wait till Google releases newer versions of airflow in cloud composer. – Gluttony

I'm using 1.10.9 version and the Conf option doesn't exist. – Intact

I'm using 1.10.10 version and the Conf option doesn't exist either – Maxfield

You people want 1.10.12 or higher ;) – Glogau

I saw the same problem of missing the conf option, and now understand why. As of 1.10.12, the conf option appears only when RBAC UI is enabled. Once I turned it on, I was able to find the field on GUI even the version is 1.10.8. airflow.apache.org/docs/stable/configurations-ref.html#rbac – Tectrix

Answer 2

With latest versions [we use 2.2.2] of Airflow we only pass config json

When we click on the Trigger/Play button available against the DAG, we get a drop down with 2 options

1. Trigger DAG
2. Trigger DAG w/ config

Clicking on the second option you can pass the conf in json format and that can be used in the script and its usage can be viewed while you create the configuration JSON as below.

To access configuration in your DAG use {{ dag_run.conf }}. As core.dag_run_conf_overrides_params is set to False, so passing any configuration here won't override task params.

Goutweed

how would you access this config in your DAG code? – Armelda

@HosseinKalbasi, #71674968 – Zebra

Answer 3

Add this line to your airflow.cfg file:

[webserver] ... show_trigger_form_if_no_params = True

Then restart the webserver.

Trioxide

Answer 4

You can pass a configuration in dictionary format while triggering a dag using dag_run.conf parameter. This configuration enables us to use a dynamic value for a variable for each run.

In the dag, the code to access a config passed via dag_run.conf is as follows:

dag_run_task = BashOperator(
    task_id= 'dag_run_conf_task',
    bash_command= "echo  The sample value given via dag_run config is  {{dag_run.conf['key1']}} ",
)

Pass the config while triggering the dag.

triggering dag with custom config

passing conf in the UI

Final restult

final log

Make sure to core.dag_run_conf_overrides_params is set to True, so passing any configuration here will override task params which can be accessed via {{ params }}.

Halfsole