Repository navigation
Closed as not planned
Description
Describe the bug, including details regarding any error messages, version, and platform.
It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read.
This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads.
My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice.
Here is where the file is opened to read in table schema.
Here is code to reproduce:
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.local import LocalFileSystem
import os
# create a fsspec filesystem to track the number of times files are opened
class LocalFSWithAccessCounter(LocalFileSystem):
def __init__(self, *args, **storage_options):
super().__init__(*args, **storage_options)
self.access_count = {}
def open(
self,
path,
mode="rb",
block_size=None,
cache_options=None,
compression=None,
**kwargs,
):
if path in self.access_count:
self.access_count[path] += 1
else:
self.access_count[path] = 1
return super().open(
path,
mode,
block_size,
cache_options,
compression,
**kwargs,
)
# create a simple test table
df = pd.DataFrame({'col1':range(2),'col2':range(2)})
out_table = pa.Table.from_pandas(df)
# write then read as single parquet file.
# file is opened twice using read_table
pq.write_table(out_table, 'example.parquet')
fs = LocalFSWithAccessCounter()
in_table = pq.read_table('example.parquet', filesystem=fs)
print('files open count:\n', fs.access_count, '\n\n')
#files open count:
# {'example.parquet': 2}
# write then read as dataset with partitions.
# one file is opened twice when using read_table
pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
fs = LocalFSWithAccessCounter()
in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs)
print('dataset file open count:\n',fs.access_count)
#dataset file open count:
# {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }
Component(s)
Python
Activity
westonpace commentedon Jun 9, 2023
Do you know which version of pyarrow you were using? There was a fix put in place for 12.0.0 (#33972) that may have addressed this.
kimotorc commentedon Jun 9, 2023
I am using 12.0.0. I read through #33972 and it seems that was an issue within the Dataset API, whereas the extra file opening seems to be initiated from the python shim.
westonpace commentedon Jun 20, 2023
Yes, however, pretty much all reads are using the dataset API internally now. The paths have been merged for simplicity of maintenance.
read_tablewill create a dataset with one file and then read it. The first time the file is opened happens when the dataset is created (to get the schema of the file). I am pretty sure it isn't actually reading the schema twice though. I think it is something like...I agree that it is somewhat less than ideal that the file is opened twice.
thisisnic commentedon Nov 18, 2026
This still happens on 26.0.1 and is expected with the current design: the first open reads the schema when the dataset is created, the second reads the data. If opens are expensive for your filesystem, passing
schema=toread_table()or usingpq.ParquetFile(...).read()both open the file once.