How to set/get airflow variables which are in json format from command line

Question

I can't edit values of airflow variables in json format through cloud shell.

I am using cloud shell to access my airflow variable params (in json format) and it gives me the complete json when i use following command:

gcloud composer environments run composer001 
--location us-east1 variables 
--get params

However I want to edit one of the values inside json, how do i access that?

I referred to the documentation and various other links on google however could only find how to set variables that are not in json format but are a single value variables.

Decompress

Answer 1

Cloud Composer CLI and Airflow CLI only operate on top-level variables, not their JSON contents.

You can use Airflow UI to edit your JSON variable, as the UI loads the whole variable and you can edit it in place. Or if you need to update a specific value inside your JSON variable through command line, you can first export your variables to a JSON file:

gcloud composer environments run \
    [ENVIRONMENT] --location [LOCATION] \
    variables -- --export /home/airflow/gcs/data/your-vars.json

gcloud composer environments storage data export \
    --environment [ENVIRONMENT] --location [LOCATION] \
    --source your-vars.json --destination .

edit the value inside JSON using a command like jq:

jq '.params.jsonkey = "newvalue"' your-vars.json > your-updated-vars.json

and import the updated file back to Cloud Composer:

gcloud composer environments storage data import \
    --environment [ENVIRONMENT] --location [LOCATION] \
    --source your-updated-vars.json

gcloud composer environments run \
    [ENVIRONMENT] --location [LOCATION] \
    variables -- --import /home/airflow/gcs/data/your-updated-vars.json

Janniejanos

Thanks. Can i just upload my params.json each time whether its edit or creation of new key inside airflow variables using the following: gcloud composer environments run \ [ENVIRONMENT] --location [LOCATION] \ variables -- --import – Decompress

Yes, variable creations and updates will work this way, as long as your params.json file represents top-level Airflow variables and you have imported this file to Cloud Composer data storage. – Janniejanos

what is the difference between the environments run and environments storage data export step? Similarly, between environments storage data import and environments run? – Rhearheba

If you wanna set a variable, without using json: "gcloud composer environments run myenv --location us variables -- set test_var 123" – Farcical