You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Unify Parquet PageIndexPolicy with ColumnChunkMask #11395
The interaction between PageIndexPolicy and the ColumnChunkMask is complicated, especially when there is an existing PageIndex in ParquetMetadata
One potential API we could consider is a new variant in PageIndexPolicy::Mask to avoid having to thread through another set of variables.
pubenumPageIndexPolicy{/// Do not read the page index.#[default]Skip,
...
/// Ensure the specified indexes exist, and error if notRequired(ColumnChunkMask),/// Load the specified indexes if they existOptional(ColumnChunkMask),}
However, that would be an API change as PageIndexPolicy isn't marked as non_exhastive so we would have to either
Wait for the next major release
Go with this API, and then unify PageIndexPolicy, and deprecate with_column_index_mask et al methods in next major version 🤔
Describe the solution you'd like
#11157 will ship with the policy and mask specified separately, so this represents option 2 above. The goal is to modify PageIndexPolicy either as above, or potentially via new variants, to provide the chunk masks along with the policy. This will be a breaking API change so it will have to wait for 61.0.0.
Describe alternatives you've considered
Keep the existing API, but that is pretty cluttered already. It's best to have a single mechanism to specify how to build the page indexes.
Activity