Python - PIP : Is it possible to specify architecture?

Question

I am building a Python Lambda function code to be executed on AWS (Linux).
My Dev machine is not Linux.
If I do pip install, I am getting Windows/MacOS related modules.

Is there a way to fetch Linux based dependencies always? Such as a CLI argument for target architecture.

Panayiotis

That's because python is supposed to be cross platform. Sadly, some dependencies use system-specific libraries and functions that limits their use on a small set of platforms (or a single one). I advise researching your libraries, see what system they're intended to be used for and replace them with others to suit your needs. – Jefferyjeffie

Thanks @catalin. I will explore this option. – Panayiotis

Answer 1

I just had the exact same problem with Lambda, Python, and trying to get Pillow to work. After much searching around, I came across this article that explained the problem, and provided a great solution.

https://medium.freecodecamp.org/escaping-lambda-function-hell-using-docker-40b187ec1e48

Basically, you can run docker on your Mac, which allows you to run a Linux instance. Then using pip, install whatever library you want to use. You can then include those files in your ZIP file and upload it to Lambda. Works like a charm.

These docker images approximate the AWS Lambda environment pretty accurately. You can use them to create packages.

Mueller

I was using the same for a while now. Forgot to update the post. Thanks @James Eberhardt – Panayiotis

Answer 2

You should be able to prefix arch -arm64 or arch -x86_64 to install modules for different architecture. For example:

arch -arm64 pip install click

Another detail example:

# Install arm64 arch:
arch -arm64 pip install cffi

# Check:
lipo -info ./.venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
Non-fat file: ./.venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so is architecture: arm64

# Removing the module:
pip uninstall cffi

# Install x86_64 arch:
arch -x86_64 pip install cffi

# Check:
lipo -info ./.venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
Non-fat file: ./.venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so is architecture: x86_64

Semele