Install newer version of sqlite3 on AWS Lambda

Question

I want to use Window functions on sqlite3 on my python3.8 code running on AWS Lambda. They are available since version 3.25.

Unfortunately, on AWS Lambda Python3.8, sqlite3 library is outdated:

>>> sqlite3.sqlite_version
'3.7.17'

while locally, on my homebrew install of Python3.8: (working)

>>> import sqlite3
>>> sqlite3.sqlite_version
'3.31.1'

How can I get an sqlite3 version > 3.25 on AWS Lambda Python 3.8 ?

Whitnell

Answer 1

I found a way: I used the external package pysqlite3, in the binary version.

in my requirements.txt

pysqlite3-binary==0.4.4

in the code

try:
    import pysqlite3 as sqlite3
except ModuleNotFoundError:
    import sqlite3  # for local testing because pysqlite3-binary couldn't be installed on macos

Whitnell

Hey Vincent. Please mark your answer as the right one. This is allowed ;) – Disservice

Hi Jens, I didn't know but "You can accept your own answer in 2 days". Also, maybe I don't have the best answer... let's check that tomorrow – Whitnell

It works fine in AWS Lambda. I can confirm that current SQLite3 version in Python 3.9 distribution on AWS Lambda is still '3.7.17' so Windows Function fails with an error like: "OperationalError: near '(': syntax error". What I don't get is why current SQLITE3 version (available) is 3.37.0 and the version used in AWS Lambda is 3.7.17 – Afteryears

Somehow for me this does not seem to work, django still complains about the old sqlite3 version. Anything that could have gone wrong? – Podium

Note that pysqlite3-binary only has x86_64 wheels. So it won't install on M1 Macs, etc. I have solved this problem using pipenv. – Purgative

Answer 2

Since I'm using the new arm64 MacBook Pro, the solution of installing pysqlite3-binary is not very feasible for me. I also had to work with a controlled pipeline that only recognizes Pipfile.lock but not the Pipfile so the other solution also doesn't work for me. Instead I found a way to compile libsqlite3.so.0 from the source code and use it to override the default one on AWS Lambda.

First you want to compile sqlite amalgamation from the source code. I created this Dockerfile so we can easily compile for linux/amd64 from arm64 machine:

FROM amazonlinux:2
RUN yum install -y tar gzip gcc make
ARG SQLITE_VERSION=3430200
RUN curl -sSLO https://www.sqlite.org/2023/sqlite-autoconf-${SQLITE_VERSION}.tar.gz
RUN tar xf sqlite-autoconf-${SQLITE_VERSION}.tar.gz; \
    mv /sqlite-autoconf-${SQLITE_VERSION} /sqlite-autoconf; \
    cd /sqlite-autoconf; \
    ./configure; \
    make

Now we can build the Docker image with command docker build --platform linux/amd64 -t sqlite3-builder -f Dockerfile .

The last make command will put the libsqlite3.so.0 under the directory /sqlite-autoconf/.libs. We can then use docker cp to extract it:

id=$(docker create sqlite3-builder)
docker cp $id:/sqlite-autoconf/.libs/libsqlite3.so.0.8.6 lib/libsqlite3.so.0
docker rm -v $id

Now we can upload this libsqlite3.so.0 file with our other code to AWS Lambda.

The last step is to set an environment variable in AWS Lambda that overrides the system SQLite3. For example, LD_PRELOAD=/var/task/lib/libsqlite3.so.0. Note that the /var/task (which is value of LAMBDA_TASK_ROOT variable), is default path to your Lambda function code.

Ointment