MWAA Airflow 2.0 in AWS Snowflake connection not showing

Question

Snowflake is not showing in the connections dropdown. I am using MWAA 2.0 and the providers are already in the requirements.txt MWAA uses python 3.7 dont know if this can be a thing

Requirements.txt:
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
asn1crypto
azure-common
azure-core
azure-storage-blob
boto3
botocore
certifi
cffi
chardet
cryptography
greenlet
idna
isodate
jmespath
msrest
numpy
oauthlib
oscrypto
pandas
pyarrow
pycparser
pycryptodomex
PyJWT
pyOpenSSL
python-dateutil
pytz
requests
requests-oauthlib
s3transfer
six
urllib3
apache-airflow-providers-http
apache-airflow-providers-snowflake
#apache-airflow-providers-snowflake[slack]
#apache-airflow-providers-slack
snowflake-connector-python >=2.4.1
snowflake-sqlalchemy >=1.1.0 \

A

This might be worth to have a look: docs.aws.amazon.com/mwaa/latest/userguide/… – Py

Answer 1

If anyone is in this trouble, instead of choosing Snowflake in the dropdown, you can choose AWS as the connection and will work fine.

A

this is a perfect answer. – Hydr

Answer 2

It took me a while to finally figure this one out after trying many different parameter combinations.

My full Snowflake URL is:

https://xx12345.us-east-2.aws.snowflakecomputing.com

The correct format for the Host field is:

xx12345.us-east-2.snowflakecomputing.com

For the Extra field, this is what worked for me:

{
    "account": "xx12345.us-east-2.aws",
    "warehouse": "my_warehouse_name",
    "database": "my_database_name"  
}

Make sure you put Amazon Web Services for the Conn Type, like @AXI said.

Also, I have these modules defined in my requirements.txt file:

apache-airflow-providers-snowflake==1.3.0
snowflake-connector-python==2.4.5
snowflake-sqlalchemy==1.2.4

My Airflow version is 2.0.2.

Palpitant

Amazon Web Services connection type includes AWS Access Key ID and AWS Secret Access Key. How do I get this from the Snowflake? – Calamitous

Answer 3

I have observed this issue in MWAA(Airflow 2.8-Python3.11) deployed in private Mode, while in Public Mode it is visible.

Nevertheless if you create the connection with AWS type, it will result in the error

airflow.exceptions.AirflowException: You are trying to use `common-sql` with AwsGenericHook, but its provider does not support it. Please upgrade the provider to a version that supports `common-sql`.

Therefore as of the time of writing, to create snowflake connections you have to use AWS Secrets Manager secret as defined here

Archaic

Answer 4

According to MWAA docs, it should be enough to add apache-airflow-providers-snowflake==1.3.0 to the requirements file. When I added it to the existing MWAA env, where I had already tried many different combinations of packages, it helped partially. It was possible to create a connection using CLI, but not with UI.

But, when I created a new clean MWAA env with the requirements file as stated in mentioned AWS doc, it worked well. The connection was available in UI.

Nsf