AWS SAM: Shared files across Lambda functions

Question

I am working with AWS SAM (Serverless Application Model) to build Python 3.6 lambda code in an API Gateway setup. As such, I have a single template.yaml file that creates several Lambda functions. They are organized with the lambda functions each in their own sub-directory within the project. The lambda also share several common files which I keep in a shared folder.

project-home
 -lambda_a_dir
   -lambda_a.py
 -lambda_b_dir
   -lambda_b.py
 -shared_dir
   -shared.py

The problem is that while Pycharm can clearly see the shared.py, SAM cannot and refuses to recognize the shared files, with the following error: Unable to import module 'lambdaA': No module named 'shared' If I move a copy of the shared.py file into each lambda directory, both Pycharm and SAM are happy and I can build/deploy to AWS.
My question: how can I build the SAM template with the shared files living in the shared directory?
So far, I have tried:

• Symbolic link and MacOS alias.
• Various combinations of CodeUri alternatives
• Local package with __init__ and setup.py. (I can't use a public package because the code is private and cannot not be put on a public repository.)

Here is my template file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  lambdaA:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ./lambda_a_dir/
      Handler: lambda_a.lambda_handler
      Runtime: python3.6
  lambdaB:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ./lambda_b_dir/
      Handler: lambda_b.lambda_handler
      Runtime: python3.6

Salamanca

You should check out AWS Lambda Layers. Putting the shared_dir content in a separate layer should do the trick. – Helio

FWIW: There is an issue and a PR sent to the AWS CLI team that the aws cloudformation package command should follow symlinks. – Athelstan

@Helio Your answer was correct. If you will write up your response as an answer, I will give you credit for it. – Salamanca

As you already have a complete answer as part of your question now, I suggest you split out this part, make an actual answer out of it and mark that as accepted answer. – Helio

Answer 1

Following the recommendation from @Dunedan I created a Layers object for each lambda function with the shared code, this effectively added those routines to the PythonPath for those functions. I also added the following to the API template definition with the new Layers properties:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  lambdaA:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ./lambda_a_dir/
      Handler: lambda_a.lambda_handler
      Runtime: python3.6
      Layers: 
        - arn:aws:lambda:us-west-1:012345678:layer:my_shared_zip:1
  lambdaB:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ./lambda_b_dir/
      Handler: lambda_b.lambda_handler
      Runtime: python3.6
      Layers: 
        - arn:aws:lambda:us-west-1:012345678:layer:my_shared_zip:1

Note that the code needs to be zipped before it is uploaded and needs to have a directory structure of the following, with the code inside a directory with the name of the language. In this case since I was using Python, the code needed to be in the python directory, and the python directory was then zipped:

my_shared_zip.zip
 -python
   -shared.py
   -other_shared.py
   -more_shared.py

Last note. While ideally, this shared-python directory should be deployed directly by the sam deploy command into the Layer objects, I have found that support for Layers in the AWS SAM CLI is still so new and so buggy, that at this point its not functional. Hopefully in the coming months it will be fixed. In the meantime, I need to manually install new versions of the shared-zip file myself. Sigh.

Salamanca

Answer 2

The layers solution looks like a hack. I tried creating symlink to "shared" folder and it worked - the shared folder was successfully packed along with my lambda function.

cd lambda_a_dir
ln -s ../shared

Carswell

I think its a question of shared code vs duplicated code. With Layers, the same layer can be attached (shared) across dozens of Lambda functions and it is easy to confirm that all of them are sharing the same layer (since Layers have versioning). With the symlink solution, each Lambda function will have its own private copy of the "shared" files, leading to the question of what code each lambda is using. That said, I'm starting to think that Containers (with ECR/ECS) may be easier to manage, but I'm still working that out. – Salamanca

100% correct! Personally, I'd prefer functions to have all required code baked inside the zip. Small drawback of "symlink" solution: chalice deploy doesn't follow symlinks :( – Carswell

This is not working for python lambda using CDK :'( – Whydah

Answer 3

have you tried using Rocketsam CLI? It solves exactly this issue (sharing code between lambdas using symlinks which are automatically created during build time). It also allows to split the YAML file so each lambda can have a separate YAML file.

Ation

Answer 4

Why not just get rid of the separate lambda_a_dir and lambda_b_dir folders and instead put the Lambda source files (lambda_a.py and lambda_b.py) into the root path, ie. under the same level as shared/? I organized my SAM repository this way, and had no issues at all accessing the shared modules between the different Lambdas.

Antipater

IMO this missed the purpose of the question - how to actually leverage the smaller bundle size of each lambda being separate whilst being able to maintain some shared code without having to bundle all lambdas together – Placement