I work in a mono repo where I have a AWS SAM (lambda) application. The structure is more or less like so:
.
├── my_folder1
├── file1.py
├── __init__.py
└── file2.py
├── my_folder2
│ └── my_folder3
│ ├── __init__.py
│ └── file3.py
├── sam-app
│ ├── events
│ │ └── event.json
│ ├── README.md
│ ├── samconfig.toml
│ ├── template.yaml
│ ├── tests
│ └── hello_world
│ ├── __init__.py
│ ├── requirements.txt
:preview:
numpy
pandas
...
│ └── app.py
:preview:
import pandas
from my_folder1.my_folder2.my_folder3.file3 import MyClass
...
.
.
.
more_folders...
.
.
.
You can separate my application into three portions:
- External pip packages like
pandas
,numpy
, etc. These packages change rarely. Occassionally I add a new one. They take a long time to install. They need to be built in a docker container which corresponds to the lambda runtime (as anyone who has used numpy/pandas has learned the hard way). Source code within the mono repo, implicitly called from my lambda. So if you are following along with the preview above.
my_folder1.my_folder2.my_folder3.file3
might look like:from my_folder1.my_folder837278 import my_utility_function class MyClass: my_utility_function() ...
This code changes often. Doesn't need to build in docker image.
- My actual lambda code explicitly called by a handler within:
./my_folder1/sam-app/hello_world/app.py
Let's say I run the sam build --use-container
command. This is great! It creates .aws-sam/build/MyFunction/pandas
, .aws-sam/build/MyFunction/numpy
, etc. And these pip packages are built against amazon-linux and can be used by the lambda. The problem is that my local imports (e.g. from my_folder1.my_folder2.my_folder3.file3 import MyClass
) will clearly fail.
My question is: Is it possible to sync the local imports of my code to my sam application using the existing template specification? Best docs regarding template spec
Additional information:
How I get around this currently? I write bash scripts that look like this and run them after
sam build --use-container
:. . . rsync "${EXCLUSIONS[@]}" -r $LOCAL_MODULES0 $LOCAL_MODULES1 ./.aws-sam/build/${FUNCTION}/my_folder2 rsync "${EXCLUSIONS[@]}" -r $LOCAL_MODULES2 $LOCAL_MODULES3 ./.aws-sam/build/${FUNCTION}/ . . .
which copy/sync the relevant local directories into the build directory to seem as if they were pip installed. Because lambda must already have that folder in the python path.
I also have one just for
./my_folder1/sam-app/hello_world/
.Things that suck:
- Because the pip packages are so large you cannot view the lambda functional code in the UI. If they were layers and were put in /opt/python instead of /var/task, this would actually be possible.
Things I want:
sam build --use-container --dependencies
- Builds external packages (things installed with pip) on docker image. Takes a long time, but only need be done once (unless removed/added dependencies).sam build
- Since--dependencies
isn't specified just mounts code within./my_folder1/sam-app/hello_world/
along with any local directories specified in template.yaml- Pre build commands in template.yaml. Improves customizability.
sam local invoke
- Should besam build && sam local invoke
- External dependencies (e.g. pip packages) should be made into layers by default. Uploaded to s3.