No module named ____ error in AWS MWAA even with requirements.txt

Question

I've uploaded my requirementst.txt to a dag and referenced it when creating my Airflow Environment. Here is what my requirements.txt looks like:

apache-airflow==1.10.12
oauth2client==4.1.3
google-api-python-client==2.2.0
Flask-AppBuilder==2.3.3
boto3==1.17.59

However, I keep getting a "No module named oauthclient" error in my airflow environment. How do I check if oauthclient is actually installed or if Airflow is not reading my requirements.txt file correctly?

Thanks in advance.

Dozier

Answer 1

I solved this by recreating my environment. It looks like AWS MWAA only runs your requirements.txt once and so any updates you make after you have made your environment will not be applied.

Dozier

it seems like this is true but it cannot be – Awe

I just spoke with AWS on this issue, looks like it just takes some time to see the changes. You should be able to see the requirements file running in the scheduler logs. – Awe

I just updated requirements.txt without recreating environment. You just need to edit the environment and select the latest version of requirements.txt file. It takes some time but it updates. – Raybourne

Answer 2

You don't need to recreate MWAA. MWAA uses versioned S3 bucket. Your requirements.txt is also versioned and stick with the old version if you don't manually make it, point to the latest version.

Just go to MWAA page, click "edit", and choose the latest version of your requirements.txt. Then it will update.

Tamar

Answer 3

Updating requirements in the MWAA is not straightforward:

• Once the requirements text is changed , you have to rebuild the env again
• Before you do this , please double check to make sure that you have selected the right version of the requirements .txt file in the environment creation page
• Most of the time the version of .txt file is the older one, you have to manually select the newest one!
• Simply editing and updating the existing environment will install the older version of the requirements.txt file!

Aerometer

Answer 4

mwaa should read the requirement.txt again, if you edit the mwaa configuration through the console and press save or from cli, run: aws mwaa update-environment --name <environment name>

Bypath

Answer 5

I was also facing the same error when I was trying to upload numpy , tensorflow and other libraries for my training task, I wasnt able to solve the problem so what I did was I just ran the pip install commands as a task in my dag. Also the version compatibility is also handled by pip

Here is the code:

Import sys
Import subprocess

@task
def installing_dependencies_using_subprocess():

    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "numpy"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "pandas"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "seaborn"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "scikit-learn"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "tensorflow"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "joblib"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "matplotlib"])
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "s3fs"])

    return {"message": "Dependencies installed successfully"}

Peers