Ansible aws_s3 module fails says Boto3 is missing when it is not

Question

This Play installs python3, pip3, boto3 and botocore, and tries to use aws_s3 module to download a file:

TASK [run yum update -y using yum module] 
**********************************************************************
ok: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install python3 and pip3] *************************************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install boto3 and botocore with pip3 module] ******************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Create a directory if it does not exist using file module] ****************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [downlod file from s3 with aws_s3 module] **********************************************************************************
fatal: [ip-10-200-2-137.us-west-2.compute.internal]: FAILED! => 
{"changed": false, "msg": "Python modules \"botocore\" or \"boto3\" 
are missing, please install both"}

It fails because it says boto3 is missing, but it actually it is not:

From the Target host you can see that boto3 was installed:

[ec2-user@ip-10-200-2-137 ~]$ pip3 freeze
boto3==1.9.120
botocore==1.12.120
docutils==0.14
jmespath==0.9.4
python-dateutil==2.8.0
s3transfer==0.2.0
six==1.12.0
urllib3==1.24.1
[ec2-user@ip-10-200-2-137 ~]

This is the task that installed boto3:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

This is the task that fails:

- name: downlod file from s3 with aws_s3 module 
    aws_s3:
      bucket: mybucket
      object: mybucket/jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get   

The target host does have two versions of Python installed:

[ec2-user@ip-10-200-2-157 ~]$ which python
/usr/bin/python
[ec2-user@ip-10-200-2-157 ~]$ which python3
/usr/bin/python3

My config file looks like this:

[defaults]
private_key_file=/home/ec2-user/manual-builds/key.pem
ansible_python_interpreter=/usr/bin/python3

Is this a bug? I see some similar questions have been asked going back almost a year, but I see no solutions - thanks much for any help.

Plasia

Answer 1

The problem was that my playbook had two tasks and Ansible was using the python2 interpreter for the first one AND the second one. The second task needed the python3 interpreter to work so I had to specifiy it at the task level:

- name: downlod file from s3 with aws_s3 module
  vars:
      ansible_python_interpreter: /usr/bin/python3    
  aws_s3:
      bucket: launch-data
      object: jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get 

Plasia

this didn't work for me, I got the same error on Amazon Linux 2 – Dow

Answer 2

Ansible use /usr/bin/python as default python interpreter. And you install AWS libraries for python3 only:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

You can install AWS libraries to python2 by using pip or install for both (python3 and python2) or you can define: ansible_python_interpreter=/usr/bin/python3 in your inventory file then you limit ansible execution to python3 only.

Seal

Thanks - Python 2 is pre-installed on amamzon Linux. If I limit ansible execution to python3 only - will all ansible modules still work? or will some fail because they need python 2? – Plasia

I added this to my cfg, but got the exact same result. My cfg contents are added to the original post – Plasia

@Plasia are you sure that pip-3.7 is the default for your system? – Seal

I'm not sure how to tell, but python2 comes pre-installed on the awslinux2 image that I use, so I would guess that Python2 is default – Plasia

@Plasia it's not on ansible.cfg, but on the inventory (hosts) file. Also, why is your answer the accepted one, when ozlevka provided the answer earlier? – Primogeniture

Thanks @Pedro Gordo, so ansible_python_interpreter=/usr/bin/python3 would go in Inventory not cfg. That’s helpful to know. I missed that in ozlevka's post, but In my situation I actually needed BOTH python versions to work with Ansible, so limiting all Tasks to a single version would not be good for me – this is why the solution of specifying version at the Task level was chosen. – Plasia

Answer 3

Ansible might not be referring to the incorrect python version. Simply run ansible version command to see where ansible is pointing to.

ansible --version
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Apr 15 2020, 17:20:14) [GCC 7.5.0]

As you see Ansibe is pointing to the Python module located at /usr/lib/python2.7/ (ansible python module location) you can see if this python version has boto3 and botocore packages are installed in /usr/lib/python2.7/dist-packages/ path.

If it's missing just install boto3 and botocore there.

Horseradish

Answer 4

When the ansible says

Failed to import the required Python library (botocore or boto3) Please read the module documentation and install it in the appropriate location.

Modules might already be installed. Confirm by python -m pip list installed command. Then open python in question directly in a terminal by typing python and try to import boto3 or botocore by import boto3. if it throws any error try to work out the error.

This answer might be useful.

Caveat

Answer 5

https://github.com/aws/aws-cli/issues/3092#issuecomment-550281243

Uninstalling python3-botocore then installing with pip3 seems to work.

Pyrethrum

Answer 6

I hit this error because on my EC2 system, there happened to be two versions of Python installed. /usr/bin/python3 and /usr/local/bin/python3. Only the latter had pip installed at all, under /usr/local/bin/pip3. So by default, somehow, Ansible was attempting to use the Python interpretter /usr/bin/python3, but this Python does not have any pip and does not have boto3 or other required libraries. However, the Ansible module for pip seemed to be resolving to /usr/local/bin/pip3 despite the fact that this did not match the Python interpreter. So when I include a task like this

  - name: Ensure botocore and boto3 modules are installed
    pip: 
      name: [ "boto3", "botocore"]
      extra_args: "--user"

Ansible returns "OK" because the pip it uses does indeed detect boto3 and botocore. Ansible is not aware that these packages are installed under the "wrong" Python which does not actually match the Python at /usr/bin/python3 which it was attempting to use for things like S3 object actions.

I think I have resolved this by including this in my playbook

- name: Run AWS Tasks
  hosts: "{{ myhosts | default('all') }}" # read from cli a single host or list of hosts
  gather_facts: yes
  vars:
    ansible_python_interpreter: /usr/local/bin/python3 

which forces Ansible to use the Python interpreter at /usr/local/bin/python3 instead of the one it mistakenly auto-detects at /usr/bin/python3.

Note that I think its somewhat precarious, and possibly dangerous, to attempt to install libraries for the Python at /usr/bin/python3 since I believe this is the "system Python". I think this is how I ended up with a separate Python at /usr/local/bin/python3 in the first place.

Also note that its helpful to debug this with a playbook task that can print out some of the internal Ansible variables to help reveal which Python executable is being used such as

  - name: Check Python version
    debug:
      msg:
        - "Python interpreter: {{ ansible_facts.python.executable }}"

Brottman