No module named 'airflow.providers.ssh' on AWS Airflow (Amazon MWAA)

Question

I need to use sshoperator in a DAG on AWS Airflow (Amazon MWAA), so I imported the following library in my DAG file

from airflow.contrib.operators.ssh_operator import SSHOperator

It seems sshoperator has been defined in paramiko library, so I have added the following modules to requiremets.txt file as well

• paramiko>=2.6.0
• pysftp>=0.2.9
• sshtunnel>=0.1.4

But still, I'm getting the following error when I browse DAGs

Broken DAG: 
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
  line from airflow.providers.ssh.operators.ssh import SSHOperator
  ModuleNotFoundError: No module named 'airflow.providers.ssh'

My underestanding was sshoperator in an essential and built in operators in Airflow, so technically I shouldn't add any extra libraries or plugins to Airflow.

Godman

Answer 1

This command fixed it for me:

pip install apache-airflow-providers-ssh

Sitka

Works great. Note that for me, does not require a restart of any of the airflow components, but did require waiting several seconds for the change to kick in. – Ethelethelbert

Ah, just noticed this though: You'll have to restart airflow to create an ssh connection according to this answer – Ethelethelbert

where do you input that command? do you have to ssh into the fargate instance for MWAA? – Showker

@Showker not sure if the self managed MWAA allows this, I believe you should have given it in requirement.txt when you were setting it up. – Sitka

Don't think this is relevant to MWAA. You don't have access to the server the airflow instance is running on. – Nance

Answer 2

Was researching something similar and believe that I found the answer to both of our questions... According to Docker Questions...

Providers package is no longer included with Airflow, but you can separately install them with pip using the specific backport package

It looks like there is a backport for ssh here. So probably just have to add that to your requirements.txt?

Spelaean

I have fixed the issue by adding apache-airflow-providers-ssh==1.3.0 to requirements.txt – Godman

the only disatvantage is that once you add extra modules via the requirements.txt, it then prevents you from running certain airflow cli commands. The worse being, you cant run a backfill – Katabatic