Getting PIL/Pillow 4.2.1 to upload properly to AWS Lambda Py3.6

Question

Background

I have been struggling for the past few days to deploy a Lambda that uses Pillow, and I am deploying using Python 3.6. It may be noteworthy also that I am developing this on a Windows 10 environment.

First Attempts

I began by having pip install my packages strictly in my workspace by doing the following:

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\pillow"

I have other packages, and tried installing the packages in the same manor, one of them specifically was praw and I did so by:

pip3 install praw -t "D:\Work and Projects\...\...\<projectdir>\praw"

After zipping the contents of my project together, I uploaded my package up to Lambda and upon my first test I received the error:

Unable to import module 'motw_lambda': cannot import name '_imaging'

I then removed the Pillow package in an attempt to see where this issue was stemming from (Pillow or praw or one of the other packages). With Pillow removed, the execution succeeded. I then removed the pillow package in my package and tried:

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\PIL"

and

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\Pillow"

But got the same error with the package '_imaging'.

Further Attempts

I then followed the directions per this resource and this. I also tried using virualenv and even lambda-uploader.

Strange enough, I get the same error! I am all out of options here, and feel either I am doing something dumb, or that this is not possible on Lambda-Python3.6 currently (Although I can't image someone else hasn't used pillow in a py3.6-lambda yet...)

Any info, help, or generic resources would be appreciated!

Hilton

Can you post the full traceback? What is trying to import '_imaging' and from where? What is the full error when attempting to import? Are you able to successfully import praw? This could be any number of issues, dynamic linking failure, pythonpath, etc... You need to provide more info. fwiw, considering you are on windows, and lambda likely uses a flavor of linux, your pillow binaries are probably compiled for windows, and are not runnable in linux – Sloane

Playing on that guess, #8614761 has some hints to how to cross compile for linux from windows – Sloane

@khazhyk, The line I quoted as the error in fact is the entire stack trace. Also, thanks for the link, I'll have a look! – Hilton

Answer 1

Finally found a pretty easy solution. The trick is to install Pillow in a directory locally, zip it up, and then create a Lambda Layer. Now, the Python version you run locally must match the Python version of the Lambda. In my case locally I have Python 3.10, and the Lambda is 3.9, so I installed 3.9 locally just to use for this.

On Mac I used homebrew to install pyenv to get 3.9, but however you want to install is fine.

    !Only needed if you do not have the correct version of Python locally!
        brew install pyenv
        pyenv install 3.9 (or whatever version)
(if you do this, then you can run 'pyenv local 3.9.16' or whatever version you installed, then 'Python3 --versions' should show that version.

For example, pyenv installed in MY_USER_DIR/.pyenv/versions/3.9.16/bin/python3 OR you can set your env to with pyenv to use the specific version.

Then cd into an empty dir, and run the following:

YOUR_LOCAL_PATH_TO_CORRECT_VERSION/python3 -m pip install \
    --platform manylinux2014_aarch64 \
    --target=./python/lib/python3.9/site-packages \
    --implementation cp \
    --python 3.9 \
    --only-binary=:all: --upgrade \
    Pillow

In the above, manylinux2014_aarch64 is for the ARM runtime. So your Lambda will have to be set to that, or find the value for the X86 version. And if you are not using 3.9, then change that to your version.

Your are now good to go to create the zip for your layer. Just run the following, and then upload the zip to a layer, and make sure you select the correct Python version and architecture for your Lambda.

in the same dir you ran the above command:
zip -r ../pillowLayer.zip .

Castano

By far the easiest solution, and I've actually done this myself once I found it. In fact, you only need to specify the platform and only-binary flags, although your solution is more generic and probably most widely applicable and easy for the common goer. Another small detail is the target location I use but thats configurable of course. Updated the accepted answer because, like I mentioned, this is what I do now. – Hilton

Answer 2

Basically, you have to compile the libraries (eg, PIL) either using Docker or, even better, an EC2 instance.

1. Launch an Docker container like this: docker run --rm -it -v "%cd%:/code" lambci/lambda:build-python3.6 sh

2. Inside there, navigate to the /code dir (cd /code), create a virtualenv (virtualenv env), activate it (source env/bin/activate) and finally, install your library (pip install pillow).

3. Once you have installed your library, you can exit the container. The secret here is to move your package library to the root folder (where your main .py file is). For example, move the folder env/lib/python3.6/site-packages/PIL to the root.

Then, zip your PIL folder together with your .py file and you're set!

Full Example:

The following example compiles and compresses PIL and other common Python libraries to run in AWS Lambda.

Dockerfile:

FROM lambci/lambda:build-python3.6
WORKDIR /code
CMD ["sh", "entrypoint.sh"]

entrypoint.sh:

#!/bin/sh

set -ex

cd /code/

if [ ! -d "PIL" ]; then
    # Create virtual env, activate it and install PIL
    virtualenv env && source env/bin/activate && pip install pillow requests

    # Copy necessary files to the root folder
    rm -f build-output.zip
    #mkdir PIL
    cp -f -r env/lib/python3.6/site-packages/PIL .
    cp -f -r env/lib/python3.6/site-packages/requests .
    
    # below are the dependencies for the requests pkg
    cp -f -r env/lib/python3.6/site-packages/urllib3 .
    cp -f -r env/lib/python3.6/site-packages/certifi .
    cp -f -r env/lib/python3.6/site-packages/chardet .
    cp -f -r env/lib/python3.6/site-packages/idna .
    
    # Remove temp files
    rm -r env
fi

# ZIP it
zip -9 build-output *.py 
zip -9 -r build-output PIL
zip -9 -r build-output requests
zip -9 -r build-output urllib3
zip -9 -r build-output certifi
zip -9 -r build-output chardet
zip -9 -r build-output idna

To build (Windows):

docker build -t build_lambda .
docker run --rm -v "%cd%:/code" build_lambda

Partan

It's been a while since I asked this, but I inevitably did spin up an EC2 instance, PIP -t installed it on the instance, zipped it, then SCP'd it over to my Windows machine. From there I unzipped, dropped into my project dir, and did the final package zipping. All works, although cumbersome. Thanks! – Hilton

Yes, I figured. I wanted to leave it here for future reference for others. Thanks! – Partan

@snakecharmerb Sorry I don't have it. I've updated the answer to include a full working example that I had. I hope that helps. – Partan

Answer 3

For anyone else also new to aws python and running into this issue, you can use the layers feature, and there are existing layers here you can link to and this worked for me.

https://github.com/keithrozario/Klayers

Pillow specifically on us-east-1:

arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-Pillow:2

Rufford

Hi. After adding the layer to serverless.yaml file I still get the error. What I did is, I removed pillow from requirements.txt, but still each time while trying to make deployment, I still see 'PIL' and 'Pillow-9.3.0.dist-info' directories are created in deployment zip files . Can you help please its not working? – Menam

Answer 4

You can use a precompiled version of the PIL available here: https://github.com/Miserlou/lambda-packages

Just extract PIL folder to the deployment package and it should work.

Hypozeugma

That is showing for python 2.7, does it still work for 3.6? – Jackscrew