Using airflow with BigQuery and cloud sdk gives error "User must be authenticated when user project is provided"

Question

I am trying to run airflow locally. My DAG has a BigQueryOperator and I want to use the cloud sdk for authentication. I run "gcloud auth application-default login" in order to get the json file with the credentials. I try to test my Dag running the command: airflow test testdag make_tmp_table 2019-02-13 I get the error message "User must be authenticated when user project is provided"

If I instead of using the cloud sdk use a service account that has admin rights to BigQuery it works, but I need to use authentication through the cloud sdk.

Does anyone know what this error message means or how I can run airflow and using the cloud sdk for authentication?

I have used the following source to try to understand how I can run airflow with BigQueryOperators locally. https://medium.com/@jbencina/local-testing-with-google-cloud-composer-apache-airflow-75d4213d2893

Orva

Answer 1

I solved it by deleting the credentials file produced when I did: gcloud auth application-default login and then recreating the file. Then it worked. So I had the right method, just that something was broken in the credentials file.

Orva

This solution was not enough for me. I solved it by deleting the line "quota_project_id": "myproject" line in the application_default_credentials.json file. I don't know why Airflow doesn't like the quota project ID key, but I tested it multiple times, and this was the problem. – Recur

@Recur there are literally no other search results on this but your trick worked for me as well. – Mckale

Answer 2

either you are not working on the right project or you don't have permissions to do this job.

what I suggest is:

check your current configuration by running:

gcloud auth list

make sure that you have the right project and the right account set if not run these commands to set them:

gcloud auth application-default login

you will be prompted for a link. follow it and enter your account. after that you will see a verification code, copy it and add it to your gcloud terminal.

next thing to do is to make sure that your account has permissions to do the job that you are trying. probably you need this role roles/composer.admin if it didn't work add the premitive role roles/editor from your IAM console. But use that premitive role only for testing purposes and it's not adviasable to use it for production level project.

Diffuser

Thank you very much. It should work as you suggested. See my answer to my own question – Orva

Answer 3

as @dlbech said:

Blockquote This solution was not enough for me. I solved it by deleting the line "quota_project_id": "myproject" line in the application_default_credentials.json file. I don't know why Airflow doesn't like the quota project ID key, but I tested it multiple times, and this was the problem

Whim