Google Cloud Composer variables do not propagate to Airflow

Question

I am trying to run the example in Google Cloud Composer documentation on and I find issues, mainly two:

• the environment variables, when created via the gcloud command line or the web interface, do not propagate to the Airflow layer, making that the DAG fails complaining "Variable gcs_bucket do not exist". If I add the variables from the Airflow UI, then it works.

• the DAG is executed correctly but at no stage do I see the group of links one would see in Airflow (Tree, Graph, ... ). Even after the execution having finished successfully.

I have checked the service accounts (created by default with the environment) have permissions to edit and update variables (Editor Role) also for access via API.

I have found the documentation is not very up-to-date and for example to create the variable I need to use:

gcloud composer environments update test-environment \ 
--location=us-central1 \
--update-env-variables=gcs_bucket=gs://airflow2

instead of what the article says.

What else shall I check? What else has changed since the doc was written?

EDIT: A related problem was described here that seems to solve the second problem.

EDIT: Interestingly enough, if I run this free lab, the second problem does not happen. The first one still does. I am comparing the roles in one and the other starting with the not related to the composer project created automatically. And there is no service account created in the format of the alternate solution.

I will appreciate any help anyone could provide.

Scissure

yep, that seems to be one of the problems I had. The thing is that when I was following the instructions, I was getting errors saying that 'variables' was not a recognized subcommand (or something like that). I have tried again this morning and it works. – Scissure

I put it inside as an answer. Please accept my answer, so this question is resolved. – Rola

Answer 1

Do you have the role 'composer.worker' assigned to your service account? With me the dags then show up in the UI.

One very important note 'environment_variables' are not the same as a Variable in airflow. So updating your environment variable will not work, although you could use a environment variable instead of airflow Variables. The Airflow Variables can indeed be assigned via the UI.

Rola

Answer 2

As of Airflow v1.10.10 (available in Composer as of Sept 17 2020), environment variables in all-uppercase that follow the format AIRFLOW_VAR_<VARIABLE_NAME> will be available to Airflow as Airflow variables named <variable_name>, in lowercase.

For example:

AIRFLOW_VAR_GCS_BUCKET="example-bucket-name"

...would result in

models.Variable.get("gcs_bucket") # --> example-bucket-name

You can also use Secret Manager to store Airflow connections, secrets, and variables now. (Documentation)

Characteristic

1.10.10 is now available as of Sept. 17, 2020. Thanks for the tip! – Duprey

Thank you very much! It solved my problem. – Shanteshantee

thanks a bunch. I could not find/ see this trivial information anywhere else. You pointing it out explicitly made all the difference. – Evertor

I could not find find this so important information in any of the pages discussing "environment variables" and "airflow config" in GCP Composer documentation pages. Thanks a lot. – Dynamoelectric

Answer 3

Following the tutorial step by step failed and that brought me to try other things. In this testing alternatives I made the mistake of confusing Composer variables with Airflow variables and that is how I phrased the problem. The underlying problem though was that the tutorial steps failed probably due to some glitch during the environment setup. Apparently there are limitations in the tool, specially if you are in Australia as it is my case, like the ones posted here or here.

Scissure