How to read parquet file with a condition using pyarrow in Python

Question

I have created a parquet file with three columns (id, author, title) from database and want to read the parquet file with a condition (title='Learn Python'). Below mentioned is the python code which I am using for this POC.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyodbc


def write_to_parquet(df, out_path, compression='SNAPPY'):
    arrow_table = pa.Table.from_pandas(df)
    if compression == 'UNCOMPRESSED':
        compression = None
    pq.write_table(arrow_table, out_path, use_dictionary=False,
                   compression=compression)


def read_pyarrow(path, nthreads=1):
    return pq.read_table(path, nthreads=nthreads).to_pandas()


path = './test.parquet'
sql = "SELECT * FROM [dbo].[Book] (NOLOCK)"

conn = pyodbc.connect(r'Driver={SQL Server};Server =.;Database = APP_BBG_RECN;Trusted_Connection = yes;')

df = pd.io.sql.read_sql(sql, conn)

write_to_parquet(df, path)

df1 = read_pyarrow(path)

How can I put a condition (title='Learn Python') in read_pyarrow method?

Sylvie

Answer 1

Filters are now available read_table

table = pq.read_table(
        df, filters=[("title", "in", {'Learn Python'}), 
                     ("year", ">=", 1950)]
    )
 

Edmund

Answer 2

This is not yet supported. We intend to develop this functionality in the future. I recommend doing the filtering with pandas after the conversion from Arrow table.

Durrett

Any update? What is the current state, on 2018-07-22? – Episcopalism

arrow.apache.org/docs/python/generated/… See filter section. Haven't tried it myself. – Cheriecherilyn