How to correctly load a gem extension in AWS Lambda

Question

I'm having trouble working through a gem load error on AWS Lambda.

{
  "errorMessage": "LoadError: libpq.so.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.4/lib/pg_ext.so",
  "errorType": "Function<Sequel::AdapterNotFound>",
  "stackTrace": [
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.4/lib/pg.rb:4:in `<top (required)>'",
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/adapters/postgres.rb:6:in `<top (required)>'",
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/database/connecting.rb:88:in `load_adapter'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/database/connecting.rb:17:in `adapter_class'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/database/connecting.rb:45:in `connect'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/core.rb:121:in `connect'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/core.rb:399:in `adapter_method'",
    "/var/task/vendor/bundle/ruby/2.5.0/gems/sequel-5.16.0/lib/sequel/core.rb:406:in `block (2 levels) in def_adapter_method'",
    "/var/task/lib/warehouse/loader.rb:5:in `connection'",
    "/var/task/lib/warehouse/loader.rb:24:in `initialize'",
    "/var/task/lib/warehouse/update.rb:43:in `new'",
    "/var/task/lib/warehouse/update.rb:43:in `block in handle'",
    "/var/task/lib/warehouse/update.rb:42:in `each'",
    "/var/task/lib/warehouse/update.rb:42:in `handle'",
    "/var/task/lambda.rb:11:in `handler'"
  ]
}

I am using the Sequel library to make a PSQL connection from AWS Lambda, but it seems that the function cannot find the so file. I have packaged the dependencies in vendor/bundle, built in Ubuntu on CodeBuild, and verified that the .so file is present in the resulting artifacts uploaded to lambda. I've also edited the $LOAD_PATH, but that doesn't seem to help.

Anyone else encountered this difficulty? Any further tips on resolving or debugging?

Harriman

A bit more detail: I printed out the contents of LOAD_PATH. I clearly see ./vendor/bundle/ruby/2.5.0/gems/pg-1.1.4/lib but the error above references /var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.4/lib/pg.rb. I think that's the same path.... but do I need to be more explicit? – Harriman

Answer 1

Do you have libpq.so.5 on your lib folder?

Your error is saying that did not find libpq.so.5 on the $PATH, in AWS Lambda the folder lib is automatically loaded on the path, thus, you just need to have this file there.

Executables created outside the lambda world do not run on Lambda, furthermore, you need to compile the executables by your own on a Lambda image. This is an example in how to do that:

Gemfile

source "https://rubygems.org"

gem "pg"
gem "mysql2"

handler.rb

require 'pg'
require 'mysql2'

def run(event:, context:)
  {
    postgres_client_version: PG.library_version,
    mysql_client_version: Mysql2::VERSION
  }
end

Dockerfile

FROM lambci/lambda:build-ruby2.5

RUN yum install -y postgresql postgresql-devel mysql mysql-devel
RUN gem update bundler

ADD Gemfile /var/task/Gemfile
ADD Gemfile.lock /var/task/Gemfile.lock

RUN bundle install --path /var/task/vendor/bundle --clean

This is going to build your image, then run it to generate the PG and MYSQL executables, then copy it to your lib folder.

build.sh

#!/bin/bash -x
set -e

rm -rf lib && rm -rf vendor && mkdir lib && mkdir vendor

docker build -t pg_mysql_layer -f Dockerfile .

CONTAINER=$(docker run -d pg_mysql_layer false)

docker cp \
    $CONTAINER:/var/task/vendor/ \
    ./

docker cp \
    $CONTAINER:/usr/lib64/libpq.so.5.5 \
    lib/libpq.so.5

docker cp \
    $CONTAINER:/usr/lib64/mysql/. \
    lib/

docker rm $CONTAINER

After running ./build.sh it is going to generate the folder lib and vendor with all you need, now you just need to deploy your lambda function.

To test locally you can run: docker run --rm -it -v $PWD:/var/task -w /var/task lambci/lambda:ruby2.5 handler.run

It is going to return something similar to this:

REF: https://www.stevenringo.com/ruby-in-aws-lambda-with-postgresql-nokogiri/

REF: https://www.reddit.com/r/ruby/comments/a3e7a1/postgresql_on_aws_lambda_ruby/

Socialism

This is not working for me with sam cli: sam build --debug -t template.yaml --use-container – Bumbling

@prcoder, please provide more information. The question here is about loading gem extensions on AWS lambda, there is no CLI involved. – Socialism

Hey Ruan, apologies. The issue was with AWS Lambda Layers with SAM. This solution doesn't work with that. I tried the regular way by putting all the required native dependencies in the code and it worked. Thanks for this solution. – Bumbling

An example code for ruby 2.7: github.com/proton/lambda-ruby-pg – Trinitrocresol

Answer 2

There are few good plugins to manage dependencies for AWS lambda. serverless-ruby-layer for ruby and serverless-python-requirements for python.

For your ruby case, you can use serverless-ruby-layer simply by adding plugin related config to your serverless.yml.

service: using-docker-yums

plugins:
  - serverless-ruby-layer

custom:
  rubyLayer:
    use_docker: true
    docker_yums:
      - postgresql-devel
    native_libs:
      - /usr/lib64/libpq.so.5

provider:
  name: aws
  runtime: ruby2.5

functions:
  hello:
    handler: handler.hello

And you need to install the plugin with the below command inside your serverless project folder,

sls plugin install -n serverless-ruby-layer

Now running sls deploy will automatically deploy the gems and libs to the layer.

Check out this example here in the docs

Km

Answer 3

Here's a solution like @ruan-carlos', but in one step and a bit easier to understand.

#!/bin/bash

set -e

cat >Dockerfile <<EOF
FROM public.ecr.aws/lambda/ruby:2.7
RUN yum install -y postgresql-libs
EOF

rm -rf lib

docker build -t pg-layer-source  .

mkdir lib

docker create -ti --name dummy pg-layer-source bash

docker cp dummy:/usr/lib64/libpq.so.5.5 lib/libpq.so.5
docker cp dummy:/usr/lib64/libldap_r-2.4.so.2.10.7 lib/libldap_r-2.4.so.2
docker cp dummy:/usr/lib64/liblber-2.4.so.2.10.7 lib/liblber-2.4.so.2
docker cp dummy:/usr/lib64/libsasl2.so.3.0.0 lib/libsasl2.so.3
docker cp dummy:/usr/lib64/libssl3.so lib
docker cp dummy:/usr/lib64/libsmime3.so lib
docker cp dummy:/usr/lib64/libnss3.so lib

docker rm -f dummy

zip -r $(date +%Y-%m-%d-%s)-lib-pg-layer.zip lib

Alephnull

Answer 4

2022 UPDATE using newer pg gem versions:

FROM public.ecr.aws/lambda/ruby:2.7

# Get PostgreSQL 10 available in amazon-linux-extras
RUN yum install -y amazon-linux-extras
RUN amazon-linux-extras enable postgresql10
RUN yum install -y postgresql postgresql-devel

ADD . ${LAMBDA_TASK_ROOT}

ENV GEM_HOME=${LAMBDA_TASK_ROOT}
RUN bundle install

CMD [ "functions.my_function" ]

You will now need to install amazon-linux-extras inside the lambda/ruby:2.7 docker image. This will make available the postgresql10 package. Enable it and then install postgresql-devel. If you try to install postgresql-devel without specifying at least postgresql9.3 or greater, then it will attempt to install PostgreSQL9.2 and the pg gem will raise an error that the version of postgres client is out of date.

Kimono