Linux pyarrow undefined symbol

Question

I am running Python 3.7.2 and using Miniconda3 to create a new environment named test-env. I have installed the pyarrow package from the default channel into this environment; however, when I try and import pyarrow, the Python interpreter gives me an undefined symbol error:

Import Error: /test-env/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev

From other threads on SO, I gather that this is due to a compiler incompatibility. I'm not a developer so I'm not sure how to fix this.

Bioenergetics

Nester, which operating system are you running on? I am seeing a similar issue on Ubuntu Linux, following updates (late last week?) to Arrow/Parquet. My hunch is that if you're also on Linux, this problem with the shared object is in the system libraries, at a lower level than pyarrow – Closeup

This looks like what happens when pyarrow is compiled with an earlier/out-of-date release of Apache Arrow C++ ? – Closeup

Answer 1

I had the same error after I installed by conda install, after I conda uninstall, I did pip install, the issue went away.

>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dv6/anaconda3/envs/spark/lib/python3.6/site- 
  packages/pyarrow/__init__.py", line 54, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
  ImportError: /home/dv6/anaconda3/envs/spark/lib/python3.6/site- 
  packages/pyarrow/lib.cpython-36m-x86_64-linux-gnu.so: undefined symbol: 
  _ZNK5arrow5Field8B5cxx11Ev                                                                         
   >>> quit()

Then I conda uninstall

(spark) dv6@dv6:~$ conda uninstall pyarrow

Then I pip install

(spark) dv6@dv6:~$ pip install pyarrow --user
Collecting pyarrow
  Downloading pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl (63.1 MB)
     |████████████████████████████████| 63.1 MB 25 kB/s
Requirement already satisfied: numpy>=1.14 in 
./anaconda3/envs/spark/lib/python3.6/site-packages (from pyarrow) (1.18.1)
Requirement already satisfied: six>=1.0.0 in 
./anaconda3/envs/spark/lib/python3.6/site-packages (from pyarrow) (1.14.0)
Installing collected packages: pyarrow
Successfully installed pyarrow-0.16.0
(spark) dv6@dv6:~$ python -c "import pyarrow"
(spark) dv6@dv6:~$

Here is my Python version

(spark) dv6@dv6:~$ python
Python 3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 21:14:29)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Salinasalinas

Answer 2

I would recommend opening a ticket at https://issues.apache.org/jira/projects/ARROW/issues/ARROW-4774?filter=allopenissues

Provide how you installed pyarrow (pip or conda), which distribution of linux. If you can reproduce this in a clean environment, even better.

Exhibitionism