How does a Dataproc Spark operator return a value and how to capture and return it

Question

How does a Dataproc Spark operator in Airflow return a value and how to capture it.

I have a downstream job which capture this result and based on returned value, I've to trigger another job by branch operator.

Heymann

I am facing exactly the same problem. – Folkrock

Answer 1

Simple answer is you can't. Dataproc jobs does not support returning any variable as well as logs back to airflow.

You can try the one workaround.

Your dataproc job will write any output in file back to airflow dga bucket so your dag can read that file loaclly ( or you can write file on temp bucket then download the file from there and read ).

Somatic