Issues Installing Python 3.x via Pyenv

Question

I have just gotten a new Macbook Air with M1 chip and I am trying to install Python 3.8.3 (or any 3.x version) via pyenv. I was able to install pyenv via Homebrew, but when I try to install a new python version I get an error like the below. I believe it is something to do with the new chip and/or the Big Sur OS. I have tried the instructions at these links and the error message is the same:

https://github.com/pyenv/pyenv/issues/1643#issuecomment-655710632

https://dev.to/kojikanao/install-python-3-8-0-via-pyenv-on-bigsur-4oee

Here's what I type into the terminal and what I get in return:

$ CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 11.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/w3/wh28vlqs2txcyl67cjcd689h0000gn/T/python-build.20201217143449.26458
Results logged to /var/folders/w3/wh28vlqs2txcyl67cjcd689h0000gn/T/python-build.20201217143449.26458.log
Last 10 log lines:
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX
make: *** No targets specified and no makefile found. Stop.

Laryngeal

Can you check the result of: brew install [email protected]? not sure about pyenv, you may use virtualenv or conda if necessary – Cajole

I hit the same issue while trying to build Python on an M1 Mac. A bug repot (bugs.python.org/issue42056) and patch already exist (see also bugs.python.org/issue41100) but it doesn't look like they've been released yet. FWIW, this is specific to building from source. Installing Python binaries (such as through brew install python) works fine. – Caisson

Answer 1

UPDATE: It looks like the patch that I refer to in the first paragraph below has now been released. While I have not yet tried it myself, I suspect the native ARM64 version of Python should install fine as a result of this.

Original answer:

I hit the same issue. As mentioned in my initial comment, a bug report and patch already exist for this in the Python codebase - though the fix is yet to be released.

In the meantime, the solution I went for is to run everything (including Homebrew) in "x86_64 mode" under Rosetta 2. To do this for any given command, prepend the command with arch -x86_64. Or see this for how to set up Terminal to run all commands under Rosetta.

Specific steps:

1. Install the x86_64 version of Homebrew, per https://mcmap.net/q/167641/-how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook:

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Once installed, ensure that you are running the x86_64 version of brew from /usr/local/bin/brew. I.e. you should get the following:

$ which brew
/usr/local/bin/brew

If you're running the arm64 version from /opt/homebrew/bin/brew you need to modify your system path so that /usr/local/bin takes precedence. FWIW, the arm64 and x86_64 Homebrew installs can coexist peacefully.

2. Install pyenv and some Python build dependencies

arch -x86_64 brew install pyenv bzip2 zlib

3. Install your desired version of Python through pyenv (per this comment that OP referenced)

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

---

Epilogue:

Although I did not follow the exact process above, I believe it reflects what's required. I skipped steps like installing the XCode Command Line Tools, which are well-documented elsewhere.

If you get errors like

zipimport.ZipImportError: can't decompress data; zlib not available

or warnings like

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

it probably means those libraries aren't actually installed properly by Homebrew. Doing e.g. brew uninstall zlib && brew install zlib took care of those for me.

Caisson

Svet, thanks for taking the time to type up this answer. After losing some hours, your method worked for me. I was able to pyenv install 3.8.3, 3.6.12 and 3.6.8. Many thanks! – Meditation

Svet, thanks. This worked for me, with only one problem. I had wasted hours trying to get a version of python before 3.9.1 installed (3.9.1 is incompatible with a library I want to use). I now have 3.8.2 installed. The one problem I encountered was a message saying 'To brew update first run git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow which I did, and then followed your steps exactly and it worked. – Stambul

"you need to modify your system path so that /usr/local/bin [brew] takes precedence" over arm64 brew is critical. (I normally have /opt/homebrew/bin earlier in the path to prefer M1 native executables.) A quick fix is to run eval $(/usr/local/bin/brew shellenv) before the pyenv install – Suribachi

this solution seems to work for me for python versions <=3.8.3, but not for newer ones. any ideas why? – Apples

Answer 2

The ARM version of Homebrew installs Pyenv fine now. Although not all versions compile on the M1. 3.7.9, 3.9.2 do compile fine, however, 3.8 (it tried 3.8.7 and 3.8.8) does not have the right 'arch' target yet.

Showplace

I only see up to version 3.9.1 in pyenv. Just curious, how did you install 3.9.2? – Phaedrus

@Rob or others who find this later... sometimes pyenv in brew lags the pyenv repo itself unfortunately. i have stopped using the brew version for this reason. you can use the pyenv automatic installer (github.com/pyenv/pyenv-installer) which instead tracks the master branch of the pyenv repo to get the latest versions. – Willawillabella

and when installed via the automatic installer, you can run pyenv update to update the repo itself. note: running pyenv update does not seem to be supported when installed via brew. – Willawillabella