AWS Lambda in Python: Import parent package/directory in Lambda function handler

Question

I have a directory structure like the following in my serverless application(simplest app to avoid clutter) which I created using AWS SAM with Python 3.8 as the runtime:

├── common
│   └── a.py
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
└── template.yaml

I would like to import common/a.py module inside the Lambda handler - hello_world/app.py. Now I know that I can import it normally in Python by adding the path to PYTHONPATH or sys.path, but it doesn't work when the code is run in Lambda inside a Docker container. When invoked, the Lambda handler function is run inside /var/task directory and the folder structure is not regarded.

I tried inserting /var/task/common, /var/common, /var and even /common to sys.path programmatically like this:

import sys
sys.path.insert(0, '/var/task/common')
from common.a import Alpha

but I still get the error:

ModuleNotFoundError: No module named 'common'

I am aware of Lambda Layers but given my scenario, I would like to directly reference the common code in multiple Lambda functions without the need of uploading to a layer. I want to have something like the serverless-python-requirements plugin in the Serverless framework but in AWS SAM.

So my question is, how should I add this path to common to PYTHONPATH or sys.path? Or is there an alternative workaround for this like [serverless-python-requirements][3] to directly import a module in a parent folder without using Lambda Layers?

Atalie

you have to create package "common" and import it. geeksforgeeks.org/create-access-python-package – Gouda

@aviboy2006 : You mean adding __init__.py? Python 3.3+ has implicit namespace packages, which means that there is no requirement anymore for adding __init__.py #37140286 As I mentioned, I am using Python 3.8. I also mentioned that I am able to import packages like this in non-lambda environment, I am asking specifically for AWS Lambda(run in a container) which is different from the normal Python execution scenario(/var/task is the execution directory when run in AWS Lambda) – Atalie

lambda works on virtual environment. /var/task/ create runtime that we will not able to get import path. Better you have create package name "common" folder name. – Gouda

I am not able to understand this: "Better you have create package name "common" folder name." Do you mean deploying the package separately because like I mentioned the common folder is already a package in Python 3.3+ and I know that Lambda is a virtual environment, that is why I want a workaround. Can you please be more elaborate about this? – Atalie

i am saying you cant use common name as folder or import. because it cant override you custom folder to virtual environment folder. you can use "shared" or something different which not comes with system keyword. – Gouda

Renamed common to alphaFolder, inserted /alphaFolder, /var/alphaFolder & /var/task/alphaFolder to sys.path in separate runs; still fails! – Atalie

Why want to add sys.path? – Gouda

sys.path or no sys.path, it doesn't matter to me whatsoever. I just want a way to import modules from the parent folder in a Lambda function handler. If you know the answer, please consider posting one and mention the details! – Atalie

Let us continue this discussion in chat. – Gouda

@SyedWaqas Since you haven't shared the template.yaml, I am not sure what you have under CodeUri. I am guessing that you have it as CodeUri: hello_world/ and that's why it isn't working because SAM would only copy the code inside the hello_world directory. To also include the common folder, you would need to change your CodeUri to CodeUri: . and then you would be able to import from common.a import Alpha. To verify that your directory structure is correct or the file is even present, add this to the top of your lambda handler import os os.system("ls /path/you/want/") – Thinker

If I am correct, you cannot include multiple folders in your CodeUri. If your case is like, you have multiple directories at root but only want a couple in the lambda, you won't be able to do that. There are workarounds though and you can have a preprocessing step that updates your directory structure before sam build. For example, copying common/ directory inside the hello_world/ one. – Thinker

how did you solve this issue, I am also facing same issue and stuck here,Please share your solution if you can – Elegist

@Md.ParvezAlam I have shared my solution below. – Atalie

Answer 1

I didn't find what I was looking for but I ended up with a solution to create a single Lambda function in the root which handles all the different API calls within the function. Yes my Lambda function is integrated with API Gateway, and I can get the API method and API path using event["httpMethod"] and event ["httpPath"] respectively. I can then put all the packages under the root and import them between each other.

For example, say I have 2 API paths /items and /employees that need to be handled and both of them need to handle both GET and POST methods, the following code suffices:

if event["path"] == '/items':
   if event["httpMethod"] == 'GET':
      ...
   elif event["httpMethod"] == 'POST':
      ...
elif event["path"] == '/employees':
   if event["httpMethod"] == 'GET':
      ...
   if event["httpMethod"] == 'POST':
      ...

So now I can have as much packages under this Lambda function. For example, the following is how repository looks like now:

├── application
│   └── *.py
├── persistence
│   └── *.py
├── models
│   └── *.py
└── rootLambdaFunction.py
└── requirements.txt

This way, I can import packages at will wherever I want within the given structure.

Atalie

Answer 2

I was having a similar issue with package dependencies and according to AWS Knowledge Center you should put all of your items at the root level:

You might need to create a deployment package that includes the function modules located in the root of the .zip file with read and execute permissions for all files.

According to https://aws.amazon.com/premiumsupport/knowledge-center/build-python-lambda-deployment-package/

Percolator