Trying to download apache airflow with pip but error pops up when "building wheels for collected packages: google-re2"

Question

I'm on MacOS Mojave, Python 3.8.3, and pip 23.2. I made a py_env, activated it, and ran

pip install apache-airflow==2.6.3' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.8.txt".

At first I had an error saying it couldn't find file "pybind11". I installed pybind11 by running python -m pip install pybind11, but now theres multiple unknown type errors and other errors.

python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [140 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-cpython-38
copying re2.py -> build/lib.macosx-10.9-x86_64-cpython-38
running build_ext
building '_re2' extension
creating build/temp.macosx-10.9-x86_64-cpython-38
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -
DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -
I/Users/chu/opt/anaconda3/include -arch x86_64 -
I/Users/chu/opt/anaconda3/include -arch x86_64 -
I/Users/chu/Desktop/Airflow_test/py_env/lib/python3.8/site-
packages/pybind11/include -
I/Users/chu/Desktop/Airflow_test/py_env/include -
I/Users/chu/opt/anaconda3/include/python3.8 -c _re2.cc -o
build/temp.macosx-10.9-x86_64-cpython-38/_re2.o -fvisibility=hidden
In file included from _re2.cc:11:
...
/Users/chu/Desktop/Airflow_test/py_env/lib/python3.8/site-
packages/pybind11/include/pybind11/detail/common.h:688:18: warning: 
alias declarations are a C++11 extension [-Wc++11-extensions]
using type = ISeq;
^
fatal error: too many errors emitted, stopping now [-ferror-
limit=]
18 warnings and 20 errors generated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a 
problem with pip.
ERROR: Failed building wheel for google-re2
Running setup.py clean for google-re2
Failed to build google-re2
ERROR: Could not build wheels for google-re2, which is required to 
install pyproject.toml-based projects

Nucleoplasm

Answer 1

I ran into a similar issue in my MacOS environment and was able to solve it by re-installing python.

As Ankit Chaurasia pointed out the issue most likely is stemming from your python version having been installed using an older version of MacOS.

For me, using pyenv, I simply installed a new python version pyenv install 3.9.6 activated it locally pyenv local 3.9.6 and created a new virtual environment with that python version python3.9 -m venv my_venv.

From there, I followed the install steps again from airflow without any issues.

pip install 'apache-airflow==2.6.3' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.9.txt"

Hope this helps.

Haifa

Thanks for the answer, I found my python is from ~/miniconda3/bin/python, after I remove the miniconda and re-install airflow, it succeeds!! – Zollverein

Answer 2

I faced a similar error.

Can you first try running the following?

pip install pybind11-global==2.10.4
pip install pyre2
pip install Cmake
pip install google-re2 --only-binary ':all:

If this throws a similar error, most likely what happened is you have upgraded to Mojave and kept your Python installed from Mavericks and kept your HOME directory with the Python version installed on Mavericks. The problem is your Python thinks it is on 10.9 and indeed there is no binary version for Python 3.9 of google-re2. It would help if you reinstalled your python.

How do you manage your Pythons and which Python version do you have?

I generally recommend reinstalling newer versions of Python as they are released and cleaning up old ones (also because the old ones on Mac might have this problem, but mainly because we test airflow on the latest available versions released by the Python team which contains the latest security and bugfixes and we had a few cases where it caused our tests to fail.

More on this can be found on this thread:Apache-airflow slack discussion

Savonarola

I tried to run the last command but I just get this error. ERROR: Could not find a version that satisfies the requirement google-re2 (from versions: none) ERROR: No matching distribution found for google-re2 – Nucleoplasm

The problem is your Python thinks it is on 10.9 and indeed there is no binary version for Python 3.9 of google-re2. Please try the following command brew install -s re2 pybind11. If this doesn't work, please reinstall your Python or anaconda setup. That helped me. – Savonarola

Did it work for you? – Savonarola

No, I deleted and redownloaded python 3.11 in my venv and made the venv my python interpreter but its still not working. Google-re2 is still missing. – Nucleoplasm

Answer 3

I thought I already answered, but I had to upgrade my MacOS then delete and redownload Conda.

Nucleoplasm

Answer 4

I tried running the command-

pip install 'apache-airflow==2.8.1' \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt" --no-cache-dir

Notice that I am using the flag --no-cache-dir which has helped me install it successfully in my virtual environment.

Westhead

Answer 5

This was solved for me upgrading pip to a latest version.

python3 -m pip install --upgrade pip

I tried all the methods mentioned above but the issue was not solved.

Lopsided