How do I read a file in the airflow cloud composer bucket?

Question

To separate bigquery queries from the actual code I want to store the sql in a separate file and then read it from the python code. I have tried to add the file in the same bucket as the DAGs and also in a sub folder, but it seems like I can't read the file when it airflow is running my python script with the sql files.

What I want is this:

gs://my-bucket/dags -> store dags
gs://my-bucket/dags/sql -> store sql files

The sql files might be files that I need to read first to inject things that is not supported by the jinja templating.

Can I do the above?

Azral

Can you clarify what you're trying to do that's not supported by the Jinja templating? Am I understanding correctly that you've created a Connection object to GCS but are getting some sort of error trying to access SQL files at the paths listed? – Component

First of, quite new with airflow and composer. What I want to do is to dynamically set which xcom variable to read. – Azral

So I need to create a gcs connection? I just assumed that composer/airflow synced the gcs bucket with the disk that airflow have access to. – Azral

Answer 1

Cloud Composer mounts the GCS bucket using a FUSE driver from gs://my-bucket to /home/airflow/gcs/. This means that the gs://my-bucket/dags folder is available in the scheduler, web server, and workers at /home/airflow/gcs/dags.

Your DAGs should be able to read the SQL files from the /home/airflow/gcs/dags/sql directory.

Note: the /home/airflow/gcs/data directory is available on workers but not the webserver.

Wyeth

Awesome. Will try first thing tomorrow. Sounds like what I need. – Azral

I'm guessing a better approach is to use the DAGS_FOLDER env. variable. – Azral

another approach might be to use settings.DAGS_FOLDER , this way you can reference the dag folder so that you do not have to hardcode the path, in local test, this path can be different and settings.DAGS_FOLDER can help you – Hoffarth