"cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" on AWS Lambda using a layer

Question

What I want to achieve

To scrape an website using AWS Lambda and save the data on S3.

The issues I'm having

When I execute Lambda, the following error message appears.

{ "errorMessage": "Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "fb66bea9-cbad-4bd3-bd4d-6125454e21be", "stackTrace": [] }

Code

The minimum Lambda code is as follows.

import requests
import boto3 

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    upload_res = s3.put_object(Bucket='horserace-dx', Key='/raw/a.html', Body='testtext')
    
    return event

An layer was added to the Lambda. Files were save in python folder using the commands below , frozen in a zip file, then uploaded to AWS Lambda as a layer.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user

• The bucket horserace-dx exists
• The folder raw exists
• The role of the Lambda is properly set. It can read from and write to S3
• The runtime of the Lambda is Python 3.9. The python version of the local computer is 3.9.13.

What I did so far

I google "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" and found some suggestions. I made the layer with the following code and tried again in vain.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
!pip install urllib3==1.26.15 -t ./python --no-user

So what should I do to achieve what I want to achieve? Any suggestions would be greatly appreciated.

Lukasz

Is there a reason you are installing that specific version of urllib? – Lymphadenitis

@Lymphadenitis on this page (qiita.com/SatoshiGachiFujimoto/items/437b0ccaba817903fb72) there was a suggestion that the same error was solved using that version. I know the author was using docker while I'm not, but I tried that anyway. – Lukasz

Try urllib3<2 – Lymphadenitis

@Lymphadenitis I got the same error – Lukasz

Answer 1

1. Execute the following commands.

   pip install requests==2.25.0 -t ./python --no-user pip install beautifulsoup4 -t ./python --no-user pip install pytz -t ./python --no-user

2. On PyPI, download the following whl files from the pages of numpy and pandas

• numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
• pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

3. Unzip the files and move the contents to the python folder.

4. Zip the python folder and upload it to AWS Lambda Layer.

5. Set the layer to the Lambda.

6. Then the code runs without errors.

Lukasz

Answer 2

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

You are encountering this issue because you’re using botocore which does not support urllib3 2.0 yet.

Since you’re deploying to AWS Lambda, you’ll need to explicitly pin to urllib3<2 in your project to ensure urllib3 2.0 isn’t brought into your environment.
(Source)

urllib3<2

Follow this guide for how to deploy Python Lambda functions with .zip file archives.

If you cannot get it to work via the .zip file, consider deploying via a container image instead by following this guide.

Periscope

We noticed this change between urllib3-2.0.3 version and urllib3-2.1.0 – Serif

Answer 3

In my case I just specified requests version (runtime python3.9) -

requests==2.28.2

and it worked.

Lamplighter

2.29.0 also worked for me the full command is pip install requests==2.29.0 – Diopside

pip install requests==2.28.2 -t ./. will install it only in your current folder – Dingus

2.31.0 didn't work. used 2.28.2 and it worked. using image with public.ecr.aws/lambda/python:3.9-x86_64. thanks for this comment @ashrafminhaj – Semi

Answer 4

In my case,I encounter this error when using transformers library. I solved this problem by running pip install 'urllib3<2'

Disraeli

this worked for me in bash: pip install 'urllib3<2' – Romanesque

Answer 5

I solved it by setting urllib3<2. There is a breaking change with urllib3 2.0.

For example, if you have a requirement.txt file, then add a line urllib3<2 to the file.

Refer to this for a better explanation: https://github.com/psf/requests/issues/6443#issuecomment-1535667256

Dartboard

this worked for me in bash: pip install 'urllib3<2' – Romanesque

Answer 6

The resolution that worked for me was to update the AWS Lambda runtime to 3.10 and if applicable any AWS Lambda layers to python 3.10 and then ensure you have packaged the latest release of requests. If you are using botocore or boto3 you will also need to ensure that you have included them in the code package or in an AWS Lambda Layer including versions boto3>=1.26.153 and botocore>=1.29.153.

Spann

That's actually a little misleading, using previous stable version of libraries did solve the issue while using python3.9 as runtime. – Lamplighter

I am using python3.8 and urllib3==1.26.18 and installing boto3==1.34.23(previously 1.19.9) solved my problem. – Naominaor

Answer 7

I solved this problem by using python 3.11 when setting up my lambda function in AWS, the lower versions tends to throw this error the more

Blueing

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Impersonality

upgrading lambda environment wasn't enough in my case--still had to rebuild package with correct requests version per @ashraf minhad – Compendious

Answer 8

This is because openai urllib3 issue

Solution : Open any folder and type this command “pip install openai requests==2.29.0 urllib3==1.26.16 -t python” this will install pip with these specific versions of requests and urllib3

Cyrilcyrill

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Impersonality

Answer 9

You can solve this adding in your deployment.txt:

urllib3==1.16

Joung

This does not work, you will get error 'is_ipaddress' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)", 1.26.16 works fine – Seibert

Answer 10

Had a similar issue, with this and other libraries. The root cause of the problem is library compatibility, so to solve it without specifying the library versions, we can use the lambda's runtime Docker image:

mkdir layer
cp requirements.txt layer/requirements.txt
docker run -ti -v $(pwd)/layer:/app -w /app --entrypoint /bin/bash public.ecr.aws/lambda/python:3.11 -c "pip3 install --target ./python -r requirements.txt"

We get at layer/python, all the libraries that we need for the layer. Then we just zip it and create the layer in AWS.

Here I'm using Python 3.11 for the lambda, but any other version should work

Note: on windows powershell, change $(pwd) to ${pwd}.

Ranaerancagua

Answer 11

In the requirements.txt add the line urllib3<2 and build and deploy the code, then it will resolve the issue. Even if requirements.txt doesnt have that please add it, because there might be some installations for which urllib3 will be a prerequisite and that will be downloaded even if its not mentioned directly.

Addle

Answer 12

In my case, the installed urllib3 was already <2. The solution was to update boto3 library to the latest version.

Smear

Answer 13

The solution I tried for the above error is creating virtual environment in python using

python -m venv <name_of_virtualenv>

and then installing boto3 in the virtual environment.

Juicy

...and did it work? Please read How to Answer. – Jonquil