Serverless deploy with serverless-python-requirements does not work

Question

I am trying to deploy a python lambda function using serverless. I need the psycopg2 library so I used the serverless-python-requirements plugin and added it to my serverless.yml file.

plugins:
  - serverless-python-requirements
custom:
    pythonRequirements:
        dockerizePip: non-linux

I can deploy the function successfully:

Serverless: Installing requirements from 
/Users/Desktop/postgresql/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (43.07 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...

..............
Serverless: Stack update finished...

But when I check my Cloudwatch logs, it says that there is "No module named 'psycopg2' ".

The structure of my zip file is:

    |--node_modules
    |--.gitignore
    |--handler.py
    |--package-lock.json
    |--package.json
    |--serverless.yml

No .serverless/ dir was created when I ran serverless deploy...

Any help would be greatly appreciated!

Dunker

Answer 1

The plugin serverless-python-requirements uses cache. Make sure to clean it before package your service. Furthermore, psycopg requires to be compiled with a lambda environment:

# serverless.yaml
custom:
    pythonRequirements:
        dockerizePip: true

# bash
$ serverless requirements cleanCache
$ serverless deploy --stage <stage>

Mnemosyne

Answer 2

Did you installed python dependencies correctly?

serverless is trying to install requirements from Serverless: Installing requirements from /Users/Desktop/postgresql/.serverless/requirements/requirements.txt

I would suggest following

• create virtualenv virtualenv ./env
• activate it source env/bin/activate
• install your depdencies pip install psycopg2
• freeze your local dependencies pip freeze > requirements.txt

Assuming your code is importing your dependency, deploy using serverless.

Cotenant

Answer 3

You only need to define your Python dependencies in the requirements.txt file. You can do this manually or use pip freeze with:

 pip freeze > requirements.txt

Chlorite

Answer 4

I deployed Python Flask application on AWS and while setting it up, got the same error. I solved it by doing the following,

I hope you are using Virtual Environment where you installed all the Python libraries you need. I didn't see virtual environment files in your list of folders. You can check the readme file of my git repo to see how to build from scratch. I documented all the steps I followed - https://github.com/shyam454/Flask_AWS

Note: The repo works but I still developing it, so the endpoints mentioned in Readme are not yet in the code. But it is still good regarding Python + Flask setup

If you already have virtual environment, do the following steps,

Before deployment using serverless, do pip freeze > requirements.txt

This command will include all the python libraries you installed in your virtual environment to be present in requirements.txt file

In the serverless.yml file, add these lines

plugins: - serverless-wsgi - serverless-python-requirements

Now do, serverless deploy

You can observe that the serverless will install all the requirements from the requirements.txt. you should see the below lines while serverless is deploying,

Serverless: Generated requirements from xxxxxx\AWS\Flask_AWS\requirements.txt in xxxxxx\AWS\Flask_AWS.serverless\requirements.txt...

Serverless: Installing requirements from xxxxxxx\AWS\Flask_AWS.serverless\requirements\requirements.txt ...

This is when it installs the python libraries in the cloud and you will not get the error no module available.

I hope this helps you.

Ouphe

Answer 5

I suggest that you use AWS lambda layer. Upload manually the working packages of psycopg2 in aws lambda layer.

and reference it on serverless.yml name: [name of your serverless project] layers: - [arn-of aws layers package]

Note: layers is on the same level of name in serverless.yml

Kinsley

Answer 6

1. first create virtual environment of python then activate or use global environment
2. install node js and install serverless-offline package (below command use)

npm install -g serverless-offline

serverless.yaml file

service: XXXX #Name of your App

plugins:
  - serverless-offline

package:
  individually: true
  patterns:
    - '!./**'

then run below command ( if you use vscode terminal then need to be administrator access to the terminal or better to use or os terminal)

serverless -offline

if you want to deploy live then create lambda layer

and use lambda layer ARN in serverless yaml file like

functions:
  prodAddFeatures:
    name: prodAddFeatures
    handler: addfeatures/lambda_function.lambda_handler
    events:
      - http:
          path: /about/addfeatures
          method: post
          cors: true
          
    layers:
      - arn:aws:lambda:us-XXXX-1:XXXXXXX:layer:final:3
    package:
      individually: true
      patterns:
        - './addfeatures/lambda_function.py'

then use

serverless deploy 

Uniflorous

Answer 7

I also run Lambdas in AWS that require psycopg2 and ran into deployment issues as well. The solution is to use aws-psycopg2 and include that in your requirements.txt

Beabeach

Answer 8

A bit late but for those who are facing the similar issue.

You are deploying your requirements as zip. You have to unzip the requirements before you import them in your python script. Add this line of code on top of your python script where you want libraries to be imported:

try:
 import unzip_requirements
except ImportError:
 pass

You can throw an error on exception if you desire or just pass.

Ginkgo