Installing Python dependencies in AWS Codestar with Lambda

Question

I'm trying out AWS Codestar. My objective is to deploy a non-trivial lambda endpoint, that is, where the handler has dependencies. Ideally, I'd like to be able to specify them in a requirements.txt file somewhere but this seems not to be so straightforward. Specifically, I would like to deploy a lambda handler that depends on nltk and where the files for the nltk tokenizer "punkt" are downloaded as part of the Codebuild process and packaged up for Lambda.

How can this be done through buildspec.yml and template.yml? Below, I'm trying to install the pip dependencies to a subdirectory lib and include this in the zip artifact.

When run, Codebuild is able to install dependencies, import nltk and run tests, the deployment to Lambda succeeds, and the right files are being packaged up in the lib subfolder (I downloaded the ZIP file to check) but I see errors in the Lambda logs: unable to import module 'index': No module named 'nltk'.

Here is my buildspec.yml:

    version: 0.2

    phases:
      install:
        commands:
          - pip install -r requirements.txt -t lib
          # Upgrade AWS CLI to the latest version
          - pip install --upgrade awscli

      pre_build:
        commands:
          - python -V
          - export PYTHONPATH=$PYTHONPATH:./lib
          - export HOME_DIR=`pwd`
          - mkdir $HOME_DIR/nltk_data/
          - export NLTK_DATA=$HOME_DIR/nltk_data
          - python -m nltk.downloader -d $NLTK_DATA punkt
          - python -m unittest discover tests

      build:
        commands:
          - aws cloudformation package --template template.yml --s3-bucket 
$S3_BUCKET --output-template template-export.yml

    artifacts:
      type: zip
      files:
        - template-export.yml
        - '**/*'

and my template.yml:

    Resources:
      HelloWorld:
        Type: AWS::Serverless::Function
        Properties:
          Handler: index.handler
          Runtime: python3.6
          Environment:
            Variables:
              PYTHONPATH: ./lib
          Role:
            Fn::ImportValue:
              !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
          Events:
            GetEvent:
              Type: Api
              Properties:
                Path: /
                Method: get
            PostEvent:
              Type: Api
              Properties:
                Path: /
                Method: post

Crybaby

Answer 1

The reason the above didn't work is that for whatever reason, PYTHONPATH doesn't work on AWS Lambda (even though it seems to work with Codebuild). The below configuration works.

buildspec.yml:

    version: 0.2

    phases:
      install:
        commands:
          - pip install -r requirements.txt -t .
          # Upgrade AWS CLI to the latest version
          - pip install --upgrade awscli

      pre_build:
        commands:
          - python -V
          - export HOME_DIR=`pwd`
          - mkdir $HOME_DIR/nltk_data/
          - export NLTK_DATA=$HOME_DIR/nltk_data
          - python -m nltk.downloader -d $NLTK_DATA punkt
          - python -m unittest discover tests

      build:
        commands:
          - aws cloudformation package --template template.yml --s3-bucket 
$S3_BUCKET --output-template template-export.yml

    artifacts:
      type: zip
      files:
        - template-export.yml
        - '**/*'

and my template.yml:

    Resources:
      HelloWorld:
        Type: AWS::Serverless::Function
        Properties:
          Handler: index.handler
          Runtime: python3.6
          Environment:
            Variables:
              NLTK_DATA: ./nltk_data
          Role:
            Fn::ImportValue:
              !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
          Events:
            GetEvent:
              Type: Api
              Properties:
                Path: /
                Method: get
            PostEvent:
              Type: Api
              Properties:
                Path: /
                Method: post

Crybaby

I see you changed -t /.lib to -t . But you also added NLTK_DATA variable to the lambda. Wasn't NLTK_DATA necessary even before you changed -t to . instead of ./lib ? – Bleachers