Occur "Could NOT find Arrow" error when using pip_pypy3 to install pyarrow

Question

I am trying to use pypy3 to install pyarrow, but some errors occur.

Basic information is blow:

macOS 10.15.7
Xcode 12.3
python version 3.7.9
pypy3 version 7.3.3 
pyarrow version 0.17.1
cmd is 'pip_pypy3 install pyarrow==0.17.1'

Some key information and error content in the log:

...
Requirement already satisfied: numpy>=1.14 in /usr/local/Cellar/pypy3/7.3.3/libexec/site-packages (from pyarrow==0.17.1) (1.19.5)
...
cmake -DPYTHON_EXECUTABLE=/usr/local/Cellar/pypy3/7.3.3/bin/pypy3  -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-install-jagh2frg/pyarrow_522bc325fbd74d9ebdf84f29e3a66c0c
...
2021-01-10T21:19:27,670   -- Found Python3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (found version "3.7.9") found components: Interpreter Development NumPy Development.Module Development.Embed
2021-01-10T21:19:27,713   -- Found Python3Alt: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
2021-01-10T21:19:28,226   CMake Warning (dev) at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
2021-01-10T21:19:28,227     The package name passed to `find_package_handle_standard_args` (PkgConfig)
2021-01-10T21:19:28,227     does not match the name of the calling package (Arrow).  This can lead to
2021-01-10T21:19:28,227     problems in calling code that expects `find_package` result variables
2021-01-10T21:19:28,227     (e.g., `_FOUND`) to follow a certain pattern.
2021-01-10T21:19:28,227   Call Stack (most recent call first):
2021-01-10T21:19:28,227     /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPkgConfig.cmake:67 (find_package_handle_standard_args)
2021-01-10T21:19:28,228     cmake_modules/FindArrow.cmake:39 (include)
2021-01-10T21:19:28,228     cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,228     CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,228   This warning is for project developers.  Use -Wno-dev to suppress it.

2021-01-10T21:19:28,229   -- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
2021-01-10T21:19:28,255   -- Could NOT find Arrow (missing: Arrow_DIR)
2021-01-10T21:19:28,255   -- Checking for module 'arrow'
2021-01-10T21:19:28,269   --   No package 'arrow' found
2021-01-10T21:19:28,270   CMake Error at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
2021-01-10T21:19:28,270     **Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR**
2021-01-10T21:19:28,270     **ARROW_FULL_SO_VERSION ARROW_SO_VERSION)**
2021-01-10T21:19:28,271   Call Stack (most recent call first):
2021-01-10T21:19:28,271     /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
2021-01-10T21:19:28,271     cmake_modules/FindArrow.cmake:412 (find_package_handle_standard_args)
2021-01-10T21:19:28,271     cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,271     CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,272   -- Configuring incomplete, errors occurred!

Do I need to add some environment variables(such as ARROW_INCLUDE_DIR) to complete the installation?

---

According to what Uwe L. Korn said, I have installed arrow c++ using brew install brew install apache-arrow and brew install apache-arrow-glib, but there is another problem.

2021-01-11T17:44:05,139   creating build/bdist.macosx-10.7-x86_64/wheel/pyarrow/include
2021-01-11T17:44:05,140   error: can't copy 'build/lib.macosx-10.7-x86_64-3.7/pyarrow/include/arrow': doesn't exist or not a regular file
2021-01-11T17:44:05,149   ERROR: Failed building wheel for pyarrow
2021-01-11T17:44:05,151 Failed to build pyarrow
2021-01-11T17:44:05,152 ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,152 Exception information:
2021-01-11T17:44:05,152 Traceback (most recent call last):
2021-01-11T17:44:05,152   File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/base_command.py", line 224, in _main
2021-01-11T17:44:05,152     status = self.run(options, args)
2021-01-11T17:44:05,152   File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/req_command.py", line 180, in wrapper
2021-01-11T17:44:05,152     return func(self, options, args)
2021-01-11T17:44:05,152   File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/commands/install.py", line 364, in run
2021-01-11T17:44:05,152     ", ".join(pep517_build_failure_names)
2021-01-11T17:44:05,152 pip._internal.exceptions.InstallationError: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,200 Removed build tracker: '/private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-req-tracker-lp8m3f4g'

Saracen

Answer 1

The answer to your compilation problem will be that you should install Arrow C++ first. This will though only bring you a single step further and you will hit a new roadblock next as pyarrow is currently not working with PyPy, only CPython. There is an open upstream issue ARROW-2651 where you can track the progress but currently it seems nobody is working on that. So in your case you can either stick to CPython or help in that issue to fix the missing bits to get it running on PyPy.

Jowett

I have installed arrow c++, but there is another problem(updated in problem description). Does this problem need pyarrow side or pypy side to solve? – Saracen

It probably needs to solved on both sides. It doesn't look that complicated but it needs someone to look into it. – Jowett

Thanks，looking forward to pyarrow supporting pypy soon – Saracen

Does it support pyston? – Slag

What does "installed arrow c++" mean in the context of Debian 10 (buster / stable)? I can not find a fitting package for that. – Gamin

Answer 2

Install Arrow C++, see documentation C++ and GLib (C) Packages for Debian GNU/Linux, Ubuntu, AlmaLinux, CentOS and Amazon Linux at https://arrow.apache.org/install/

Herwin

Answer 3

This is what worked for me.

see: https://apache.jfrog.io/artifactory/arrow/

and do something like this for your version:

wget https://apache.jfrog.io/artifactory/arrow/ubuntu/apache-arrow-apt-source-latest-jammy.deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-jammy.deb
sudo apt update
sudo apt install -y -V libarrow-glib-dev # For GLib (C)
sudo apt install -y -V libarrow-dev # For C++

Also, I downgraded from python 3.11.0rc2 to 3.10.8

And then re-ran pip install

The error came from trying to install some stuff for dbt:

snowflake-connector-python[secure-local-storage]<2.8.2,>=2.4.1

Sedimentology