import pyarrow not working <- error is "ValueError: The pyarrow library is not installed, please install pyarrow to use the to_arrow() function."

Question

I have tried installing it in the terminal and in juypter lab and it says that it has been successfully installed but when I run df = query_job.to_dataframe() I keep getting the error " ValueError: The pyarrow library is not installed, please install pyarrow to use the to_arrow() function.". I have no idea how to fix this. Any advice? I am trying to access data from google data studio ultimately with the code,

from google.cloud import bigquery
import pandas
import numpy
import pyarrow
bigquery_client = bigquery.Client()
import os 
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] ='full file path here.json'
QUERY = """
SELECT * 
FROM `warehouse`
LIMIT 100
"""
query_job = bigquery_client.query(QUERY)
df = query_job.to_dataframe()

Spelter

Hello, can you share your requirements.txt ? – Oxytocin

did you try updating all your packages to latest version? – Aeolotropic

I am still having this problem as well. – Deuno

Answer 1

I got the same error message ModuleNotFoundError: No module named 'pyarrow' when testing your Python code. This behavior disappeared after installing the pyarrow dependency with pip install pyarrow.

Edit: It worked for me once I restarted the kernel after running pip install pyarrow

Flagellum

Answer 2

I had the same issue. Fixed after the following:

pip install --upgrade 'google-cloud-bigquery[bqstorage,pandas]'

Source: https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas

Fleck

Nope. Didnt fix for me. – Stratocumulus

Answer 3

I had the same issue because I had pyarrow 2.0, however you will need version 1.0.1 . Try running this line: pip install pandas-gbq==0.14.0

Disarrange

Answer 4

To avoid using fetch_pandas_all(), I have used fetchall, and then converted result to pandas DataFrame I have used:

requirements.txt

snowflake-connector-python==2.4.3
pandas==1.2.4

dag.py

    def execute(self, **kwags):
        """
        :param kwargs: optional parameter. Can be used to provide task input context
        :return: returns query result in json format
        """

        ctx = snowflake.connector.connect(
            user=self.SNOWFLAKE_USER,
            password=self.SNOWFLAKE_PASSWORD,
            account=self.SNOWFLAKE_ACCOUNT
        )
        cs = ctx.cursor()
        try:
            cs.execute(self.sql_query)
            data = cs.fetchall()
            df = pd.DataFrame(data)
            print(f'\nQUERY RESULT: \n' \
                      f' {tabulate(df, headers="keys", tablefmt="psql", showindex="always")} \n')
        finally:
            cs.close()
        ctx.close()
        logging.info("Query executed successfully")
        return json.loads(data)

Shulamite

Answer 5

I have experienced a similar problem but then I used the pandas Dataframe method:

client = bigquery.Client()
try:
    df = client.query(query)
    df = pd.Dataframe(df)
except ValueError:
    print("google services not available or invalid credentials.")

df.head()

Counterattraction

Answer 6

I had the same issue, but when writing to BQ from pandas using client.load_table_from_dataframe().

Changing around some package versions solved the issue (not sure if these all played a role or not):

pyarrow: 14.0.2

google-cloud-bigquery: 3.25.0 -> 3.15.0

pandas: 2.2.2 -> 2.1.4

Parnell

Answer 7

just need to install pyarrow using pip

df = client.query(query1).to_dataframe()
 data = df.to_json()
        
 print(data['total_transactions'][0])
 print(data['total_visits'][0])

Magnitude

this answer code only includes the copy of the post authors code. it does not show the solution. Please, add the underlying pip command – Corey