pysftp library not working in AWS lambda layer

Question

I want to upload files to EC2 instance using pysftp library (Python script). So I have created small Python script which is using below line to connect

pysftp.Connection(
    host=Constants.MY_HOST_NAME,
    username=Constants.MY_EC2_INSTANCE_USERNAME,
    private_key="./mypemfilelocation.pem",
)
some code here .....
pysftp.put(file_to_be_upload, ec2_remote_file_path)

This script will upload files from my local Windows machine to EC2 instance using .pem file and it works correctly.

Now I want to do this action using AWS lambda with API Gateway functionality.

So I have uploaded Python script to AWS lambda. Now I am not sure how to use pysftp library in AWS lambda, so I found solution that add pysftp library Layer in AWS lambda Layer. I did it with

pip3 install pysftp -t ./library_folder

And I make zip of above folder and added in AWS lambda Layer.

But still I got so many errors like one by one :-

No module named 'pysftp'

No module named 'paramiko'

Undefined Symbol: PyInt_FromLong

cannot import name '_bcrypt' from partially initialized module 'bcrypt' (most likely due to a circular import)

cffi module not found

I just fade up of above errors I didn't find the proper solution. How can I can use pysftp library in my AWS lambda seamlessly?

Gipson

Answer 1

I build pysftp layer and tested it on my lambda with python 3.8. Just to see import and basic print:

import json
import pysftp

def lambda_handler(event, context):
    # TODO implement
    print(dir(pysftp))
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

I used the following docker tool to build the pysftp layer:

• https://github.com/lambci/docker-lambda

So what I did for pysftp was:

# create pysftp fresh python 3.8 environment
python -m venv pysftp

# activate it
source pysftp/bin/activate

cd pysftp

# install pysftp in the environemnt
pip3 install pysftp  

# generate requirements.txt
pip freeze > requirements.txt

# use docker to construct the layer
docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.8 python3.8 -m pip --isolated install -t ./mylayer -r requirements.txt

zip -r pysftp-layer.zip .

And the rest is uploading the zip into s3, creating new layer in AWS console, setting Compatible runtime to python 3.8 and using it in my test lambda function.

You can also check here how to use this docker tool (the docker command I used is based on what is in that link).

Hope this helps

Antigone

also You want to make sure your .zip follows this folder structure when unzipped python/lib/python3.6/site-packages/{LibrariesGoHere}. Upload that zip, make sure the layer is added to the Lambda function and you should be good to go. This is the structure that has worked for me. Thanks you so much for this help. You save my week. – Gipson

I also have some questions like how to use .pem file in AWS lambda? and while uploading file to EC2 how to use temp location of AWS lambda? – Gipson

pem is an important file. Can consider storing it as a secret string in Parameter Store, or in Lambda Environment variables (event encrypted with encryption helpers). Temp location is /tmp with max size of 500MB. Use it as you would on a normal instance. If have specific questions/issues maybe better to make new question. – Antigone

I got an issue with the zip missing a hidden file see (#51219596). And also place the contents of mylayer into a folder called python then rezip. – Agitate

not working in MAC, I build the layer in EC2, then it is working, error was in "bcript" – Lovage

@Lovage Thanks for letting me know. If the answer was helpful, its upvote would be appreciated. – Antigone

Answer 2

I still had some issues with the above accepted solution on Python 11 runtime, and changing the version of cryptography and the brcrypt packages based on this reply did the trick for me. After I installed pysftp, I ran these two:

pip install cryptography==3.4.8
pip install bcrypt==3.2.2

Brigitte