Can not import modules from Lambda Layers with Serverless framework and TypeScript
Asked Answered
V

2

6

I have several functions in my serverless app. Two of them are for REST endpoints and one is SQS handler. They all are using the same libraries. So, I want to move them to Lambda Layer and share across functions to reduce size.

I'm using Serverless framework 2.46, TypeScript 4.3 and NodeJS 14. I have the following project structure:

/
 - layers/
   - nodejs/
     - node_modules/
     - package.json
 - src/
   - handlers/ - here are my handlers 
   - etc...

I've configured TypeScript to import libraries from the layer folder like this import middy from '/opt/nodejs/@middy/core';. Here is my tsconfig

{
  "compilerOptions": {
    "preserveConstEnums": true,
    "strictNullChecks": true,
    "sourceMap": true,
    "allowJs": false,
    "target": "ES2020",
    "module": "CommonJS",
    "outDir": ".build",
    "moduleResolution": "node",
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "lib": [
      "ES6",
      "ES2019",
      "ES2020"
    ],
    "baseUrl": ".",
    "paths": {
      "/opt/nodejs/*": [
        "layers/nodejs/node_modules/*"
      ]
    }
  },
  "exclude": [
    "node_modules",
    "opt/nodejs/node_modules"
  ]
}

And have serverless config like this

service: my_serverless-app
frameworkVersion: '2'
useDotenv: true
variablesResolutionMode: 20210326
configValidationMode: error

custom:
  stage: ${opt:stage, self:provider.stage}
  dbHost:
    local: ${env:DB_HOST, ''}
    dev: ${ssm:DB_HOST_DEV}

  dbPort:
    local: ${env:DB_PORT, ''}
    dev: ${ssm:DB_PORT_DEV}

  dbUser:
    local: ${env:DB_USER, ''}
    dev: ${ssm:DB_USER_DEV}

  dbPassword:
    local: ${env:DB_PASSWORD, ''}
    dev: ${ssm:DB_PASSWORD_DEV}

  dbName:
    local: ${env:DB_NAME, ''}
    dev: ${ssm:DB_NAME_DEV}

provider:
  name: aws
  region: us-east-1
  stage: dev
  runtime: nodejs14.x
  lambdaHashingVersion: 20201221
  environment:
    NODE_PATH: "./:opt/nodejs/node_modules"
    DB_HOST: ${self:custom.dbHost.${self:custom.stage}}
    DB_PORT: ${self:custom.dbPort.${self:custom.stage}}
    DB_USER: ${self:custom.dbUser.${self:custom.stage}}
    DB_PASSWORD: ${self:custom.dbPassword.${self:custom.stage}}
    DB_NAME: ${self:custom.dbName.${self:custom.stage}}

plugins:
  - serverless-plugin-typescript
  - serverless-offline

functions:
  getLedgerRecords:
    handler: src/handlers/ledger.ledgerRecords
    events:
      - http:
          path: /ledger-records
          method: get
    layers:
      - { Ref: CommonLibsLambdaLayer }

  getLedgerRecord:
    handler: src/handlers/ledger.ledgerRecord
    events:
      - http:
          path: /ledger-records/{id}
          method: get
    layers:
      - { Ref: CommonLibsLambdaLayer }

layers:
  CommonLibs:
    path: layers/nodejs
    description: "Common dependencies"
    compatibleRuntimes:
      - nodejs14.x

When I run the app locally via command serverless offline --stage local I have no error, but when I execute an REST endpoint (or any other) I have the following error:

[offline] Loading handler... (D:\Projects\services\.build\src\handlers\ledger)
[offline] _____ HANDLER RESOLVED _____
offline: Failure: Cannot find module '/opt/nodejs/@middy/core'
Require stack:
- D:\Projects\services\.build\src\handlers\ledger.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\handler-runner\in-process-runner\InProcessRunner.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\handler-runner\in-process-runner\index.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\handler-runner\HandlerRunner.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\handler-runner\index.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\LambdaFunction.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\LambdaFunctionPool.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\Lambda.js
- D:\Projects\services\node_modules\serverless-offline\dist\lambda\index.js
- D:\Projects\services\node_modules\serverless-offline\dist\ServerlessOffline.js
- D:\Projects\services\node_modules\serverless-offline\dist\index.js
- D:\Projects\services\node_modules\serverless-offline\dist\main.js
- D:\Projects\services\node_modules\serverless\lib\classes\PluginManager.js
- D:\Projects\services\node_modules\serverless\lib\Serverless.js
- D:\Projects\services\node_modules\serverless\scripts\serverless.js
- D:\Projects\services\node_modules\serverless\bin\serverless.js

Also, I have the same problem when I'm trying to deploy the app. What am I doing wrong? Please drop me a link for tutorial how to configure lambda layers properly. Thanks in advance!

Vitric answered 21/7, 2021 at 6:53 Comment(0)
A
7

your layer configuration is correct from the Serverless Framework and TypeScript perspective.

the problem could be in the packing of the project itself (e.g. internal of serverless-plugin-typescript)

i would suggest trying another TypeScript plugin, like serverless-esbuild

using your tsconfig.json example and samples from serverless.yml. I created an example here:

https://github.com/oieduardorabelo/2021-07-21-serverless-typescript-layers

it is using esbuild for packing and transpile TypeScript to JavaScript and it is working as expected

Arbitrage answered 21/7, 2021 at 12:13 Comment(6)
I'm truly grateful for the answer. It seems like it should work. But I still have a few problems. My app is using TypeORM for Postgres DB. And It uses pg-native under the hood. When I'm trying to run it, I have an error Could not resolve "pg-native" (mark it as external to exclude it from the bundle, or surround it with try/catch to handle the failure at run-time). So now I'm trying to figure out how to mark this dependency as an external.Vitric
It seems like I made a bit of progress. I marked native modules as external esbuild: bundle: true minify: false external: [ 'pg-native', 'electron' ] These native dependencies are coming from other dependencies I'm using. But now I have another error when I start the app Serverless: Packing external modules: pg-native, electron Error Error: npm install failed with code 1 at ChildProcess.<anonymous> (/home/banha/projects/mg-services/node_modules/serverless-esbuild/dist/utils.js:48:24) at ChildProcess.emit (events.js:375:28)Vitric
hey, @IvanBanha any progress here? would be possible to share a repo with a reproducible error? i could try it from my endArbitrage
Hi @oieduardorabelo. Now unfortunately. I couldn't find any solution. Here is my repository. github.com/ivan-banha/lambda-layer-native-modules-ts I've deleted all redundant code, so there is only two REST endpoints now with middy and typeorm. I would be very grateful if you could help me with this task. I have already spent more than a week.Vitric
@IvanBanha using pg-native it's an edge case which requires some magic both using webpack and esbuild either, most probably you have to use esbuild plugin github.com/Knowre-Dev/esbuild-plugin-ignore, you can use the plugin with the serverless-esbuild like demonstrated in the example github.com/floydspace/serverless-esbuild/tree/master/examples/…Decarlo
thank a lot @oieduardorabelo. I tried with the latest serverless and still work wellMiler
T
0

I strongly suspect you've run into this issue here https://github.com/serverless/serverless/issues/10326

Which looks like a bug in the plugin which should go away with version 3 (which is currently in preview)

Although as mentioned here I ran into another bug with the 3.0.0-pre.82c9bc17 pre-release where it produces this error:

Error: EISDIR: illegal operation on a directory, unlink '/home/richard/cogcred/api-text_search/.build/node_modules'

which I worked around with this ugly hack:

sed -i 's/fs.unlinkSync(outModulesPath)/fs.removeSync(outModulesPath)/g' node_modules/serverless-plugin-typescript/dist/src/index.js
Terle answered 27/10, 2023 at 22:44 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.