Including local source code directories and files in aws-sam?
Asked Answered
C

0

6

I work in a mono repo where I have a AWS SAM (lambda) application. The structure is more or less like so:

.
├── my_folder1
    ├── file1.py
    ├── __init__.py
    └── file2.py
    ├── my_folder2
    │   └── my_folder3
    │       ├── __init__.py
    │       └── file3.py
    ├── sam-app
    │   ├── events
    │   │   └── event.json
    │   ├── README.md
    │   ├── samconfig.toml
    │   ├── template.yaml
    │   ├── tests
    │   └── hello_world
    │       ├── __init__.py
    │       ├── requirements.txt
                :preview:
                numpy
                pandas
                ...
    │       └── app.py
                :preview:
                import pandas
                from my_folder1.my_folder2.my_folder3.file3 import MyClass
                ...

    .
    .
    .
    more_folders...
    .
    .
    .

You can separate my application into three portions:

  1. External pip packages like pandas, numpy, etc. These packages change rarely. Occassionally I add a new one. They take a long time to install. They need to be built in a docker container which corresponds to the lambda runtime (as anyone who has used numpy/pandas has learned the hard way).
  2. Source code within the mono repo, implicitly called from my lambda. So if you are following along with the preview above. my_folder1.my_folder2.my_folder3.file3 might look like:

    from my_folder1.my_folder837278 import my_utility_function
    
    class MyClass:
        my_utility_function()
    
    ...
    

    This code changes often. Doesn't need to build in docker image.

  3. My actual lambda code explicitly called by a handler within: ./my_folder1/sam-app/hello_world/app.py

Let's say I run the sam build --use-container command. This is great! It creates .aws-sam/build/MyFunction/pandas, .aws-sam/build/MyFunction/numpy, etc. And these pip packages are built against amazon-linux and can be used by the lambda. The problem is that my local imports (e.g. from my_folder1.my_folder2.my_folder3.file3 import MyClass) will clearly fail.

My question is: Is it possible to sync the local imports of my code to my sam application using the existing template specification? Best docs regarding template spec

Additional information:

  • How I get around this currently? I write bash scripts that look like this and run them after sam build --use-container:

    .
    .
    .
    rsync "${EXCLUSIONS[@]}" -r $LOCAL_MODULES0 $LOCAL_MODULES1 ./.aws-sam/build/${FUNCTION}/my_folder2
    rsync "${EXCLUSIONS[@]}" -r $LOCAL_MODULES2 $LOCAL_MODULES3 ./.aws-sam/build/${FUNCTION}/
    .
    .
    .
    

    which copy/sync the relevant local directories into the build directory to seem as if they were pip installed. Because lambda must already have that folder in the python path.

    I also have one just for ./my_folder1/sam-app/hello_world/.

  • Things that suck:

    • Because the pip packages are so large you cannot view the lambda functional code in the UI. If they were layers and were put in /opt/python instead of /var/task, this would actually be possible.
  • Things I want:

    • sam build --use-container --dependencies - Builds external packages (things installed with pip) on docker image. Takes a long time, but only need be done once (unless removed/added dependencies).
    • sam build - Since --dependencies isn't specified just mounts code within ./my_folder1/sam-app/hello_world/ along with any local directories specified in template.yaml
    • Pre build commands in template.yaml. Improves customizability.
    • sam local invoke - Should be sam build && sam local invoke
    • External dependencies (e.g. pip packages) should be made into layers by default. Uploaded to s3.
Cupreous answered 17/4, 2020 at 2:3 Comment(3)
sam init --runtime python3.7?Fumy
Changing the runtime is unrelated :(Cupreous
docs.aws.amazon.com/lambda/latest/dg/configuration-layers.htmlOracle

© 2022 - 2024 — McMap. All rights reserved.