Installing Cryptography on an Apple Silicon M1 Mac

Question

Help! I'm trying to install cryptography on my m1. I know I can run terminal in rosetta mode, but I'm wondering if there is a way not to do that.

Output:

    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/[email protected]/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/tmpl4sga84k
         cwd: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-install-jko4b562/cryptography_7b1bbc9ece2f481a8e8e9ea03b1a0030
    Complete output (55 lines):
    
        =============================DEBUG ASSISTANCE=============================
        If you are seeing a compilation error please try the following steps to
        successfully install cryptography:
        1) Upgrade to the latest pip and try again. This will fix errors for most
           users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        2) Read https://cryptography.io/en/latest/installation.html for specific
           instructions for your platform.
        3) Check our frequently asked questions for more information:
           https://cryptography.io/en/latest/faq.html
        =============================DEBUG ASSISTANCE=============================
    
    Traceback (most recent call last):
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 44, in <module>
        setup(
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 432, in __init__
        _Distribution.__init__(self, {
      File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 708, in finalize_options
        ep(self)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 715, in _finalize_setup_keywords
        ep.load()(self, ep.name, value)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "src/_cffi_src/build_openssl.py", line 77, in <module>
        ffi = build_ffi_for_binding(
      File "src/_cffi_src/utils.py", line 54, in build_ffi_for_binding
        ffi = build_ffi(
      File "src/_cffi_src/utils.py", line 74, in build_ffi
        ffi = FFI()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
      Referenced from: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
      Expected in: flat namespace
     in /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so

I've tried to build and run like their instructions say in that code block to the same error. I've looked around and nobody has seemingly found the fix yet, but those things are two months old usually. What am I missing?

Romans

Answer 1

This issue is due to a mismatch between the libffi header version and the version of libffi the dynamic linker finds. In general it appears users encountering this problem have homebrew libffi installed and have a Python built against that in some fashion.

When this happens cffi (a cryptography dependency) compiles, but fails at runtime raising this error. This should be fixable by passing the right path as a linker argument. To reinstall cffi you should pip uninstall cffi followed by

LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:

This is an ugly workaround, but will get you past this hurdle for now.

Update: I've uploaded arm64 wheels for macOS so the below compilation is no longer required if your pip is up-to-date. However, if, for some reason you wish to compile it yourself:

LDFLAGS="-L$(brew --prefix [email protected])/lib" CFLAGS="-I$(brew --prefix [email protected])/include" pip install cryptography

Update 2023: As of 2023 there are wheels for cffi and cryptography for Apple Silicon. If you are still seeing this type of error, there are a few likely culprits:

1. Your pip is out of date. Update to latest! You can (and should) do all of this in a virtual environment.
2. You are attempting to install an older version of cryptography. Any version >= 3.4.6 has an apple silicon wheel (either arm64 or universal2), but we only support the latest release so always update.
3. You are attempting to install an older version of cffi. cffi cannot use abi3 so new wheels must be uploaded for each major Python release. If you're attempting to run an older cffi on a newer Python then a wheel may not be available and you'll need to compile.

Agent

Thanks, found this elsewhere but it is definitely the correct answer. – Romans

When I run pip uninstall cffi it just tells me that package isn't installed. But I'm getting the exact error above. – Theoretics

Update not working yet even though I have up to date pip installed. Still the first workaround worked. Thankyou – Abie

The updated info is for the cryptography package, which has substantially more build requirements. The primary instructions are for working around cffi problems, which (as of June 2021) does not ship a macOS arm64 wheel. Once you have that then the cryptography wheel will work and simplify your installation. – Agent

The cryptography I have on my M1 Mac's built in python doesn't work right. For example fabric won't work with it. I was able to get a python2 for arm from the Nix package manager (the only place I could find packages for python2 for the M1 Macs) and from there I installed a cryptography that works. See #68293559 – Ashliashlie

See cryptography.io/en/latest/installation/… – Collagen

This can fix a lot of cffi dependency libraries installation issues. – Clash

Thank you! I was breaking my head to resolve this issue, this command saved me hours LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all: I am using Mac M2 machine – Jenisejenkel

Answer 2

I'm using Macbook Pro M1 2020 model and faced the same issue. The issue was only with my cffi and pip versions maybe. Because these 4 steps helped me -

1. Uninstalling old cffi pip uninstall cffi
2. Upgrading pip python -m pip install --upgrade pip
3. Reinstalling cffi pip install cffi
4. Intalling cryptography pip install cryptography

Alkane

Answer 3

A little late to the party, but the solutions above didn't work for me. Paul got me on the right track, but my problem was that pyenv used the mac libffi for its build and cffi used the homebrew version. I read this somewhere, can't claim this unique insight.

My solution was to ensure that my python (3.8.13) was built by pyenv using the homebrew libffi by ensuring correct headers libraries and package config:

export LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib -L$(brew --prefix [email protected])/lib -L$(brew --prefix libffi)/lib"
export CPPFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix [email protected])/include -I$(brew --prefix libffi)/include"
export PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig:$(brew --prefix libffi)/lib/pkgconfig"

rebuilding python...

pyenv uninstall 3.8.13
pyenv install 3.8.13

killing the pip cache

pip cache purge

and, finally, reinstalling my dependencies using pipenv

pipenv --rm
pipenv sync --dev

After these steps, I was free from the dreaded

ImportError: dlopen(/private/var/folders/k7/z3mq67_532bdr_rcm2grml240000gn/T/pip-build-env-apk5b25z/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_prep_closure'

Fret

Thanks for the type, the CFLAGS and PKGCONFIG in this answer did the trick – Cardie

Wish I could give a beer for that answer, @Matt. Most of other answers I was seeing around this issue weren't going to work for me, and I had my suspicions that this was the cause, but didn't realize that pyenv would rebuild the old Python version I needed with the right linked libraries in this manner. Thank you! – Norval

I virtually accept that beer @Norval – Fret

Answer 4

Probably, you'll have a problem with more packages and each has it's own solution for Apple Silicon, it's exhausting.

I came to final solution: using x86_x64 Homebrew which installs x86 packages, including Python. Thus, all your requirements are installing as on the x86_x64 macs and there are no more problems with the compilation errors and so on.

Instructions:

1. Run iTerm2 (or default Terminal app) under Rosetta 2 (right click on the app icon -> Get info -> Open using rosetta).
2. Install homebrew as usual /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" or you can get this link from https://brew.sh/ for security reasons (never copy curl commands from stackoverflow without double-checking).
3. Add an alias in your ~/.zshrc (if you're using ZSH) or ~/.bash_profile (if you're bash user): alias brew='arch -x86_64 /usr/local/bin/brew'.
4. Turn off Open using rosetta in iTerm2 Get info.

Now, every time you'll print brew in terminal apps you'll run x86_x64 Homebrew. And when you install any package from brew, it'll work under Rosetta 2 automatically.

Ulm

Haven't gotten to this yet, but THANK YOU! This is crazy. I bought a MacMini M1 with the full intention of using it as a build server. Oye Vie! What a mess. – Theoretics

Answer 5

This answer here worked like a charm! @paveldroo

As an extension to the answer above, I went ahead and saved the alias in step 3 as alias ibrew='arch -x86_64 /usr/local/bin/brew' at ~/.zshrc

This means when I install anything with brew command, it gets installed for M1 architecture, and when I install with ibrew command it gets installed for -x86_64 architecture.

As a consequence, I installed two instances of python3 at my system one at /opt/homebrew/bin/python3 using brew and the other at /usr/local/bin/python3 using ibrew

The two versions adds some flexibility on creating the project virtual environments as needed. For example you could create virtual environments using:

1. /usr/local/bin/python3 -m venv venv for -x86_64 architecture
2. /opt/homebrew/bin/python3 -m venv venv for M1 architecture

Tye

Answer 6

I have uninstalled older version of cffi and cryptography,

pip uninstall cffi
pip uninstall cryptography

and updated the requirements.txt file from exact versions to updated versions

# requirements.txt

cffi>=1.15.1 
cryptography>=38.0.1

(version number can be different). This resolved my issue

Superstratum

Answer 7

I wasn't able to previously install cffi, until I discovered an unrelated issue. I was at this for about two days, until I found this command:

python3 -m ensurepip --upgrade

Magically, everything started working for me. It came from an issue between Python and Pip coming from different sources.

Answer stolen from this question: using pip3: module "importlib._bootstrap" has no attribute "SourceFileLoader"

Edit: This may be a courtesy of the above poster, so could be unrelated. If so, thank you anonymous human!

Olivero

Answer 8

I've fixed my issue by using the intel64 version of python. So I installed python using the macOS 64-bit universal2 installer from python website.

Then I just needed to create the virtual env using e.g: python3.11-intel64 file from /usr/local/bin/ and installed all packages with that version:

$ venv/bin/python-intel64 -m pip install -r requirements.txt

And everything started to work without any issues.

Soiree

Answer 9

It is most probably incompatibility issue, for me I resolved it by installing a stable version from cryptography:

1. pip uninstall cryptography
2. pip install cryptography==3.4.8

Continuant