Skip to content

parquet.read_table opens files twice #35894

Closed as not planned
Closed as not planned

Description

@kimotorc

Describe the bug, including details regarding any error messages, version, and platform.

It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read.

This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads.

My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice.

Here is where the file is opened to read in table schema.

Here is code to reproduce:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.local import LocalFileSystem
import os

# create a fsspec filesystem to track the number of times files are opened
class LocalFSWithAccessCounter(LocalFileSystem):
    def __init__(self, *args, **storage_options):
        super().__init__(*args, **storage_options)
        self.access_count = {}

    def open(
        self,
        path,
        mode="rb",
        block_size=None,
        cache_options=None,
        compression=None,
        **kwargs,
    ):
        if path in self.access_count:
            self.access_count[path] += 1
        else:
            self.access_count[path] = 1

        return super().open(        
            path,
            mode,
            block_size,
            cache_options,
            compression,
            **kwargs,
        )
        

# create a simple test table
df = pd.DataFrame({'col1':range(2),'col2':range(2)})
out_table = pa.Table.from_pandas(df)

# write then read as single parquet file.
# file is opened twice using read_table
pq.write_table(out_table, 'example.parquet')
fs = LocalFSWithAccessCounter()
in_table = pq.read_table('example.parquet', filesystem=fs)
print('files open count:\n', fs.access_count, '\n\n')
#files open count:
# {'example.parquet': 2} 

# write then read as dataset with partitions.
# one file is opened twice when using read_table
pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
fs = LocalFSWithAccessCounter()
in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs)
print('dataset file open count:\n',fs.access_count)
#dataset file open count:
# {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }

Component(s)

Python

Activity

  1. westonpace commented on Jun 9, 2023

    @westonpace
    Member

    Do you know which version of pyarrow you were using? There was a fix put in place for 12.0.0 (#33972) that may have addressed this.

  2. kimotorc commented on Jun 9, 2023

    @kimotorc
    Author

    I am using 12.0.0. I read through #33972 and it seems that was an issue within the Dataset API, whereas the extra file opening seems to be initiated from the python shim.

  3. westonpace commented on Jun 20, 2023

    @westonpace
    Member

    it seems that was an issue within the Dataset API

    Yes, however, pretty much all reads are using the dataset API internally now. The paths have been merged for simplicity of maintenance. read_table will create a dataset with one file and then read it. The first time the file is opened happens when the dataset is created (to get the schema of the file). I am pretty sure it isn't actually reading the schema twice though. I think it is something like...

    • Create dataset
      • open file
      • read metadata
      • close file
    • Read dataset
      • open file
      • read data
      • close file

    I agree that it is somewhat less than ideal that the file is opened twice.

  4. thisisnic commented on Nov 18, 2026

    @thisisnic
    Member

    This still happens on 26.0.1 and is expected with the current design: the first open reads the schema when the dataset is created, the second reads the data. If opens are expensive for your filesystem, passing schema= to read_table() or using pq.ParquetFile(...).read() both open the file once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions