Can't install pyarrow on OSX / Python 3.9: is this me or an incompatible package?

Question

I'm trying to install pyarrow with pip3 on OSX 11.0.1, and getting error messages.

I'm using Python 3.9 and not sure if that is the problem.

Here is the error summary:

  ERROR: Command errored out with exit status 1:
   command: /usr/local/opt/[email protected]/bin/python3.9 /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/mk/2fgx_1s96zjd1r9xzhs2ht_00000gn/T/pip-build-env-pev1z3i2/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
       cwd: None

Here is the complete (very long) error output, I'm not sure how to read this to find the source of the problem: https://pastebin.com/RQcEuwbz

It seems like perhaps pyarrow had problems with Python 3.9, but these should now have been fixed? I'm a bit stumped about what to do next.

UPDATE: Ah, found a PR to make pyarrow work with Python 3.9 that hasn't been merged yet - so I guess the pip version of pyarrow doesn't work with Python 3.9 and that is the problem. It looks like the PR hasn't been merged because it doesn't work on Windows, but that doesn't matter to me since I'm on OSX. Is there a way I could tell pip to install pyarrow from the version in the PR?

Begat

I tried pip3 install git+https://github.com/terencehonles/arrow/tree/arrow-102242.git@fcb561bf960f23d3c1b85c791bb2acff8e2575a6 but that gives me ERROR: Command errored out with exit status 128. Also I'm not sure if this is a good idea! – Begat

Answer 1

As of today Dec. 16 2020, neither fastparquet nor pyarrow have wheels for Python 3.9

This will hopefully change soon.

Grimaldi

PyArrow issues for linux wheels: issues.apache.org/jira/browse/ARROW-10224, macOS: issues.apache.org/jira/browse/ARROW-10904, Windows: issues.apache.org/jira/browse/ARROW-10905 – Mansard

If you use a conda environment, you can use pyarrow using Python 3.9. It is just that wheels are not available. – Lampas

Answer 2

Update 2021: Pyarrow has had wheels for Python3.9 since Feb/2021, Pyarrow Version 3.0.0

See Pyarrow Github

I just had to upgrade the pip version.

(Edit queue of the previous answer is full so I am posting an answer myself)

Marx

Answer 3

From the linked error logs one can see that the build fails due to numpy not being able to be installed. Thus you first need to get a running numpy installation before you can proceed with pyarrow.

      12 warnings and 1 error generated.
      error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/[email protected]/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/common -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/npymath -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/common -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/npymath -c numpy/random/mtrand/mtrand.c -o build/temp.macosx-10.15-x86_64-3.9/numpy/random/mtrand/mtrand.o -MMD -MF build/temp.macosx-10.15-x86_64-3.9/numpy/random/mtrand/mtrand.o.d" failed with exit status 1

Lampas

Thanks. I already have a working copy of numpy, though: $ pip3 install numpy produces Requirement already satisfied: numpy in /usr/local/lib/python3.9/site-packages (1.19.4). – Begat

Try using --no-use-pep517 --no-build-isolation as flags to pip3 install. This should then use the numpy from the installed environment. – Lampas

Thanks. Unfortunately that just gives me a different error pastebin.com/raw/tKz4SDN8 – Begat

There are no wheels for Python 3.9 & macOS yet, either use Python 3.8 or build from source as you are doing. In your case you are missing the C++ part of Arrow, you can get that via brew install apache-arrow – Lampas

Thank you for continuing to help! I ran brew install apache-arrow, now I get a different error instead: error: can't copy 'build/lib.macosx-11-x86_64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file. Full output pastebin.com/raw/uSmx2K1t – Begat