Skip to content

Unify Parquet PageIndexPolicy with ColumnChunkMask #11395

Description

@etseidl

Is your feature request related to a problem or challenge?

This is a followup to a comment in #11157.

The interaction between PageIndexPolicy and the ColumnChunkMask is complicated, especially when there is an existing PageIndex in ParquetMetadata

One potential API we could consider is a new variant in PageIndexPolicy::Mask to avoid having to thread through another set of variables.

pub enum PageIndexPolicy {
   /// Do not read the page index.
   #[default]
   Skip,
...
   /// Ensure the specified indexes exist, and error if not
   Required(ColumnChunkMask),
   /// Load the specified indexes if they exist
   Optional(ColumnChunkMask),
}

However, that would be an API change as PageIndexPolicy isn't marked as non_exhastive so we would have to either

  1. Wait for the next major release
  2. Go with this API, and then unify PageIndexPolicy, and deprecate with_column_index_mask et al methods in next major version 🤔

Describe the solution you'd like

#11157 will ship with the policy and mask specified separately, so this represents option 2 above. The goal is to modify PageIndexPolicy either as above, or potentially via new variants, to provide the chunk masks along with the policy. This will be a breaking API change so it will have to wait for 61.0.0.

Describe alternatives you've considered

Keep the existing API, but that is pretty cluttered already. It's best to have a single mechanism to specify how to build the page indexes.

Additional context

No response

Activity

  1. self-assigned this
    on Oct 6, 2026
  2. added theissue type on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions