Import libraries in lambda layers

Question

I wanted to import jsonschema library in my AWS Lambda in order to perform request validation. Instead of bundling the dependency with my app , I am looking to do this via Lambda Layers. I zipped all the dependencies under venv/lib/python3.6/site-packages/. I uploaded this as a lambda layer and added it to my aws lambda using publish-layer-version and aws lambda update-function-configuration commands respectively. The zip folder is name "lambda-dep.zip" and all the files are under it. However when I try to import jsonschema in my lambda_function , I see the error below -

from jsonschema import validate

{
  "errorMessage": "Unable to import module 'lambda_api': No module named 'jsonschema'",
  "errorType": "Runtime.ImportModuleError"
}

Am I missing any steps are is there a different mechanism to import anything within lambda layers?

Buckhound

Answer 1

You want to make sure your .zip follows this folder structure when unzipped

python/lib/python3.6/site-packages/{LibrariesGoHere}.

Upload that zip, make sure the layer is added to the Lambda function and you should be good to go.

This is the structure that has worked for me.

Coldhearted

Thanks I'll try this! – Buckhound

The zip's name is lambda_dependencies. So when I unzipped it created this folder structure - python/lib/python3.6/site-packages/{LibrariesGoHere}. However I still see the issue. Am I missing something? – Buckhound

This works , I changed the folder name to python3.7 since I am using python 3.7 in lambda. Thank you! – Buckhound

Sweet! Yea I would have mentioned to make sure you use the correct python version but in your original question it mentioned python 3.6 so figured that's what you were on. Congrats on getting it working. Also thank you, I can finally comment on things now lol. I've been trying to get an accepted answer forever because of the limitations put on my account until i had reputation points. – Coldhearted

Great! I hope your answers more people like me :) is this way of packaging lambda layers documented anywhere ? I was curious if I was missing to lookup any documentation that is available. – Buckhound

I had looked at a few things but at 1:42 of this video is what finally made me understand it youtu.be/ebhcs-9FYJA?t=102 – Coldhearted

Thank you very much – Buckhound

This solved my problem. I could not find this solution anywhere else or in AWS's docs. – Futurism

Glad to hear it @user3454396, I wish they wrote it out as clearly as this, I looked all around and found a video on youtube that briefly showed the proper folder structure for the various languages. Not very clear in AWS documentation – Coldhearted

According to the video they should go in either /opt/python or /opt/python/lib/python3.6/site-packages. The video also gives the indication that if you're using 3.7 or above you have to use the second one but I tried it in just /opt/python with 3.8 and it seemed to work. I feel like for simplicity it would be easier to just put in /opt/python but if you're making a layer for multiple versions then maybe you would want to do the second. – Bellbird

Here is docs about directory layout: docs.aws.amazon.com/lambda/latest/dg/… – Toothache

This also helped me: youtube.com/watch?v=3BH79Uciw5w. The zip file needs to be NAMED EXACTLY: python.zip. – Stockstill

This solution worked for me as compared to other recommended ones. For Windows, 1) Use conda 2) install python 3.7 – Incumbent

I missed 5 minutes deadline to edit and provide comprehensive - This solution worked for me as compared to other recommended ones. For Windows, 1) Use conda 2) Create environment with python 3.7 3) activate <env> 4) pip install pymysql -t . 5) zip it. Please note it is likely you will have change the path to suit suggested python/lib/python3.6/site-packages/{LibrariesGoHere}. – Incumbent

Per AWS documentation, requirements have been changed to simply be placed in a /python directory without the rest of the directory structure. aws.amazon.com/premiumsupport/knowledge-center/… – Myel

Note: The files need to be in a folder named "python" prior to zipping – Fieldpiece

Answer 2

(Update from previous answer): Per AWS documentation, requirements have been changed to simply be placed in the

/python/{LibrariesGoHere}

directory without the rest of the directory structure.

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-import-module-error-python/

Be sure your unzipped directory structure has libraries within a /python directory.

Myel

Answer 3

Here the script that I use to upload a layer:

#!/usr/bin/env bash

LAYER_NAME=$1 # input layer, retrived as arg
ZIP_ARTIFACT=${LAYER_NAME}.zip
LAYER_BUILD_DIR="python"

# note: put the libraries in a folder supported by the runtime, means that should by python

rm -rf ${LAYER_BUILD_DIR} && mkdir -p ${LAYER_BUILD_DIR}

docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.6 python3.6 -m pip --isolated install -t ${LAYER_BUILD_DIR} -r requirements.txt

zip -r ${ZIP_ARTIFACT} .

echo "Publishing layer to AWS..."
aws lambda publish-layer-version --layer-name ${LAYER_NAME} --zip-file fileb://${ZIP_ARTIFACT} --compatible-runtimes python3.6

# clean up
rm -rf ${LAYER_BUILD_DIR}
rm -r ${ZIP_ARTIFACT}

I added the content above to a file called build_layer.sh, then I call it as bash build_layer.sh my_layer. The script requires a requirements.txt in the same folder, and it uses Docker to have the same runtime used for Python3.6 Lambdas. The arg of the script is the layer name.

After uploading a layer to AWS, be sure that the right layer's version is referenced inside your Lambda.

Ster

Thank you very much @Ster – Buckhound

Does anyone know if they'll support python3.9? – Kanpur

Answer 4

There is an easier method. Just install the packages into a python folder. Then install the packages using the -t (Target) option. Note the "." in the zip file. this is a wild card.

mkdir lambda_function
cd lambda_function
mkdir python
cd python
pip install yourPackages -t ./
cd ..
zip /tmp/labmda_layer.zip .   

The zip file is now your lambda layer.

The step by step instructions includeing video instructions can be found here.

https://geektopia.tech/post.php?blogpost=Create_Lambda_Layer_Python

Nonesuch

That is exactly the same as the script that I posted. But you also have to consider that pip install needs to run in the right OS. If you build a layer in OSx with pandas, it won't work. – Ster

I like this answer because it seems fairly independent of the python version being used. (Not sure if it would work to just take guitarhero23s answer above and replace 3.6 with 3.8, for example). I just used this with 3.8 and it works great though. – Pentachlorophenol

Good answer. Adding "-t ." after pip install <packagename> is extremely important and the layer won't work if you install without adding it. – Stegall

Answer 5

In my case the issue was related to the Python version. While zipping the package I used python\lib\ python3.8\site-packages whereas i was using Python 3.9 in lambda runtime setting. So make sure you are using the same version while packaging and configuring aws lambda runtime.

Lambertson