lib-sodium/pynacl errors while building arm64-Python images for Python/Django
Asked Answered
H

1

6

Summary: I am trying to build an ARM64 image for my project to be able to run it faster locally on my macbook M1 (pro) machine, instead of using normal x64 images which run extremely slow (long startup time, long response time for local dev HTTP server, long test running times).

I have the following docker file:

FROM arm64v8/python

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    curl \
    libpq-dev \
    build-essential \
    libssl-dev \
    libffi-dev

# Install Node.js for ARM64
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash -
RUN apt-get install -y nodejs

# Set the working directory in the container
WORKDIR /app

# Install Python dependencies
RUN pip install --no-cache-dir cryptography
RUN pip install --no-cache-dir pyNaCl==1.3.0

# Copy the requirements.txt file to the container
COPY requirements.txt .

# Install additional Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the Django project files to the container
COPY . .

# Expose the Django development server port
EXPOSE 8000

# Start the Django development server
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

requirements.txt:

autopep8==1.6.0
kombu>=5.0.0  # celery dependency
celery==5.0.0
dj-database-url==0.5
django==3.2.13
django-bootstrap4==2.2.0
django-cors-headers==3.4.0
django-debug-toolbar==3.2.1
django-phonenumber-field==4.0.0
django-safedelete==0.5.6
djangorestframework==3.13.0
djangorestframework-xml==2.0.0
django-model-utils==4.2.0
gunicorn==20.0.4
ipdb==0.13.3
jedi==0.18.1  # ipython dependency
ipython==7.32.0
ipython-genutils==0.2.0
newrelic==7.16.0.178
pep8==1.7.1
python-dateutil>=2.8.0
pytz>=2019.2
redis==3.5.3
django-redis==4.12.1
requests==2.27.1
requests-mock>=1.6.0
selenium==3.141.0
twilio==7.8.0
django-filter==2.4.0
psycopg2==2.9.3
django-session-security==2.6.6
django-axes==5.4.1
beautifulsoup4==4.9.1
git+https://github.com/onitsoft/uphold-sdk-python.git#egg=uphold
isort==5.2.0
freezegun>=0.3.1
suds-community>=1.1.0
drf-extensions>=0.5.0
requests-cache==0.5.2
xmltodict==0.12
python-bitcoinrpc==1.0
urllib3==1.26.5
pillow==9.1.0
git+https://github.com/onitsoft/django-loginurl.git#egg=django-loginurl
django-countries==7.3.2
tornado==6.0.4 # flower dependency
flower==0.9.5
django-guardian==2.4.0
dredd-hooks==0.2.0
django-fsm==2.8.0
git+https://github.com/onitsoft/python-bittrex.git#egg=python-bittrex
cached-property==1.5.1
web3==5.25.0
# pycrypto==2.6.1
pycryptodome==3.14.1
django-extensions==3.1.5
django-oauth-toolkit==1.7.1
iptools==0.7.0
django-otp==0.9.3
qrcode==6.1
git+https://github.com/onitsoft/django-audit-log.git@django3-support
rlp==1.2.0
django-newsletter==0.9.1
git+https://github.com/onitsoft/[email protected]
django-email-log==1.2.0
django-nested-admin==3.4.0
#bintrees==2.0.7
sortedcontainers==2.2.2
git+https://github.com/onitsoft/OrderBook.git@5dbd18b4534ac4945b7e53315c65e1d49d9e340d#egg=orderbook
django-picklefield===2.1.1
pdbpp==0.10.2
pyyaml==5.4.1
git+https://github.com/onitsoft/ripple-python
django-constance==2.9.1
django-admin-rangefilter==0.8.4
toml==0.10.1
phonenumbers==8.12.7
django-post-office==3.4.1
slackclient==2.7.3
django-intercom==0.1.3
django-ipware==3.0.7
django-enum-choices==2.1.3
git+https://github.com/onitsoft/[email protected]
simplejson==3.17.2
django-polymorphic==3.1.0
retrying==1.3.3
ccxt==3.0.51
django-admin-multiple-choice-list-filter==0.1.1
celery-once==3.0.1
bidict==0.21.2
fireblocks-sdk==1.19.0
pycountry==20.7.3
cbor==1.0.0
git+https://github.com/onitsoft/django-admin-comments.git
lxml==4.9.1
supervisor==4.2.4
django-admin-csvexport==1.11
rsa==4.9
pydantic==1.10.2
XlsxWriter==3.0.8

Traceback: https://pastebin.com/1xtwdfXY (too long to paste here)

I have already tried everything you can find online, including:

  • Installing lib-sodium via apt instead of via pip
  • Installing version 1.3.0 of pynacl specifically using pynacl==1.3.0
  • Installing lib-sodium locally on the host machine (MacOs) via brew install libsodium

But sadly nothing helps. Sources:

Haggle answered 20/6, 2023 at 11:27 Comment(1)
In this case, I highly recommend to you install the project locally out of the Docker. It will speed up things.Vocal
S
2

PyNaCl v1.5.0 is the only version compatible with Mac ARM64.
This was released in Jan 7, 2022 - PyNaCl v1.5.0 Changelog.

Also version of libsodium provided by the distribution for the ARM64 docker image will have to be used, not the version bundled with PyNaCl v1.5.0.

So the following changes would be required for your Dockerfile:

RUN SODIUM_INSTALL=system pip install --no-cache-dir pyNaCl==1.5.0

The requirement for PyNaCl v1.5.0 has knock on effects on the dependencies declared within requirements.txt.

Here are a few that was found when building the Docker image with PyNaCl v1.5.0:

[email protected] - only compatible with PyNaCl 1.3.0
    updating to v 8.2.1 which compiled with PyNaCl 1.5.0 had knock on effects
    with the following:
    urllib3==1.26.5 -> 1.26.7
    toml==0.10.1 -> 0.10.2

N.B. Making the above changes may not allow your project to run - there obviously could be impact on the project code etc.

Saree answered 23/6, 2023 at 13:18 Comment(2)
I specified PyNaCl==1.5.0 and made the other version bumps you've suggested; build still fails with pretty much the same error (lib-sodium related). I have posted an issue in libsodium as per the instructions in the traceback github.com/yaml/pyyaml/issues/724Haggle
Wrong issue link, correct one: github.com/jedisct1/libsodium/issues/1302Haggle

© 2022 - 2024 — McMap. All rights reserved.