Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
1c77984
allow for serialization of custom page index providers
etseidl Sep 8, 2026
b2d3a1a
clippy
etseidl Sep 8, 2026
0638b87
add test of using custom page index provider with row selection
etseidl Sep 9, 2026
da1ff75
refactor and add usage stats
etseidl Sep 18, 2026
60e4b4c
Merge branch 'main' into page_index_serialization
etseidl Sep 21, 2026
a39dd8f
add test of using custom page index provider with row selection
etseidl Sep 9, 2026
30631fa
refactor and add usage stats
etseidl Sep 18, 2026
5458721
allow for serialization of custom page index providers
etseidl Sep 8, 2026
979141a
clippy
etseidl Sep 8, 2026
3a8ee6a
add plumbing for page index selective population
etseidl Sep 19, 2026
1df722e
more tests
etseidl Sep 19, 2026
9eacc74
rename PageIndexSelection to ColumnChunkMask
etseidl Sep 20, 2026
69ab61a
docs and cleanup
etseidl Sep 20, 2026
cf9d819
Merge branch 'main' into page_index_roundtrip
etseidl Sep 21, 2026
b7b529d
more miri friendly
etseidl Sep 21, 2026
b682ebe
rework tests to do no file i/o
etseidl Sep 21, 2026
bd587b2
apply suggestions from review
etseidl Sep 23, 2026
6a6e62f
Merge remote-tracking branch 'origin/main' into page_index_serialization
etseidl Sep 23, 2026
c5350f5
Merge branch 'page_index_serialization' of github.com:etseidl/arrow-r…
etseidl Sep 23, 2026
997f184
remove assert
etseidl Sep 23, 2026
0969c23
apply suggestion from review
etseidl Sep 23, 2026
a3d54e1
Merge branch 'main' into page_index_roundtrip
etseidl Sep 23, 2026
d93796e
add another assert
etseidl Sep 23, 2026
2a2fd09
clippy
etseidl Sep 23, 2026
c687e6e
restore old page index location behavior
etseidl Sep 23, 2026
55eb967
fix docs
etseidl Sep 23, 2026
92fb20c
honor preserve_page_index_locations in write_x_indexes
etseidl Sep 23, 2026
e63ad47
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 23, 2026
e31d45b
Merge branch 'page_index_serialization' into page_index_selection
etseidl Sep 23, 2026
2489741
implement review suggestions
etseidl Sep 23, 2026
8351703
clippy
etseidl Sep 23, 2026
76ec6bf
Merge remote-tracking branch 'origin/main' into page_index_roundtrip
etseidl Sep 23, 2026
57d7030
tests now pass
etseidl Sep 23, 2026
ac9a08e
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 23, 2026
49e6c76
Merge branch 'page_index_roundtrip' into page_index_selection
etseidl Sep 23, 2026
82087c5
apply suggestion from review
etseidl Sep 23, 2026
7b30fa0
revert most changes to push decoder
etseidl Sep 24, 2026
3cefc59
address a number of issues found in review, primarily documentation
etseidl Sep 24, 2026
4ee4125
modify range_for_page_index to take masks
etseidl Sep 24, 2026
f6a8fde
more efficient page index parser traversal
etseidl Sep 24, 2026
682edbf
address StatisticsConverter misalignment
etseidl Sep 24, 2026
18f34a8
enable C3 test
etseidl Sep 24, 2026
1ec36bb
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 24, 2026
8169311
fix C11: preserve existing indexes across multiple parse calls
etseidl Sep 24, 2026
09337c8
Fix page index loading with suffix prefetch
etseidl Sep 24, 2026
5b9c07d
add plumbing for page index selective population
etseidl Sep 19, 2026
1c30c6a
more tests
etseidl Sep 19, 2026
3177084
rename PageIndexSelection to ColumnChunkMask
etseidl Sep 20, 2026
942f2bf
docs and cleanup
etseidl Sep 20, 2026
199a41d
rework tests to do no file i/o
etseidl Sep 21, 2026
3dfeeca
apply suggestion from review
etseidl Sep 23, 2026
2696260
revert most changes to push decoder
etseidl Sep 24, 2026
e303c53
address a number of issues found in review, primarily documentation
etseidl Sep 24, 2026
9672e90
modify range_for_page_index to take masks
etseidl Sep 24, 2026
1c5a135
more efficient page index parser traversal
etseidl Sep 24, 2026
2c597fb
address StatisticsConverter misalignment
etseidl Sep 24, 2026
134a3b9
enable C3 test
etseidl Sep 24, 2026
dc23d99
fix C11: preserve existing indexes across multiple parse calls
etseidl Sep 24, 2026
78917ed
empty_index no longer needs pub(crate)
etseidl Sep 25, 2026
fe9af28
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
64a34bf
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
063a084
Merge branch 'page_index_selection' of github.com:etseidl/arrow-rs in…
etseidl Sep 25, 2026
b99f1bf
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
9fd6d8a
Revert "fix C11: preserve existing indexes across multiple parse calls"
etseidl Sep 25, 2026
bc975c4
replace path for C11
etseidl Sep 25, 2026
0a9dab9
restore previous compatibility exception for PageIndexPolicy::Required
etseidl Sep 25, 2026
372d065
remove mention of the C11 tag
etseidl Sep 25, 2026
3ab2fd4
finish docs for parse_page_index
etseidl Sep 28, 2026
c47bb75
import ProjectionMask
etseidl Sep 28, 2026
b9fd884
address nit
etseidl Sep 29, 2026
ad86518
replace boxed iterator with more targeted concrete iterator impl
etseidl Sep 29, 2026
4823984
revert to using single passed in Bytes
etseidl Sep 29, 2026
b920667
clean up documentation for ColumnChunkMask
etseidl Oct 1, 2026
e2211f4
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Oct 1, 2026
58ef396
fix test
etseidl Oct 1, 2026
117f3ad
mention cheap cloneability for ColumnChunkMask
etseidl Oct 1, 2026
e17b71f
clean up the chunk api and docs
etseidl Oct 1, 2026
b7033a6
add TODOs for followup
etseidl Oct 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions parquet/benches/metadata.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ use std::sync::Arc;

use parquet::basic::{Encoding, PageType, Type as PhysicalType};
use parquet::file::metadata::{
ColumnChunkMetaData, FileMetaData, LevelHistogram, PageEncodingStats, ParquetMetaData,
ParquetMetaDataOptions, ParquetMetaDataReader, ParquetMetaDataWriter, ParquetStatisticsPolicy,
RowGroupMetaData,
ColumnChunkMask, ColumnChunkMetaData, FileMetaData, LevelHistogram, PageEncodingStats,
PageIndexPolicy, ParquetMetaData, ParquetMetaDataOptions, ParquetMetaDataReader,
ParquetMetaDataWriter, ParquetStatisticsPolicy, RowGroupMetaData,
};
use parquet::file::statistics::Statistics;
use parquet::file::writer::TrackedWrite;
Expand Down Expand Up @@ -191,6 +191,28 @@ fn criterion_benchmark(c: &mut Criterion) {
})
});

let metadata = ParquetMetaDataReader::new()
.with_page_index_policy(PageIndexPolicy::Skip)
.parse_and_finish(&data)
.unwrap();
let mut reader = ParquetMetaDataReader::new_with_metadata(metadata.clone())
.with_page_index_policy(PageIndexPolicy::Required);
c.bench_function("read page index", |b| {
b.iter(|| {
reader.read_page_indexes(&data).unwrap();
})
});

let mut reader = ParquetMetaDataReader::new_with_metadata(metadata)
.with_page_index_policy(PageIndexPolicy::Required)
.with_column_index_mask(ColumnChunkMask::columns([0]))
.with_offset_index_mask(ColumnChunkMask::columns([0, 1, 4]));
c.bench_function("read page index reduced columns", |b| {
b.iter(|| {
reader.read_page_indexes(&data).unwrap();
})
});

let meta_data = get_footer_bytes(data.clone());
let options = ParquetMetaDataOptions::new().with_encoding_stats_as_mask(false);
c.bench_function("decode parquet metadata", |b| {
Expand Down
83 changes: 79 additions & 4 deletions parquet/src/arrow/arrow_reader/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ use crate::column::page::{PageIterator, PageReader};
use crate::encryption::decrypt::FileDecryptionProperties;
use crate::errors::{ParquetError, Result};
use crate::file::metadata::{
PageIndexPolicy, ParquetMetaData, ParquetMetaDataOptions, ParquetMetaDataReader,
ParquetStatisticsPolicy, RowGroupMetaData,
ColumnChunkMask, PageIndexPolicy, ParquetMetaData, ParquetMetaDataOptions,
ParquetMetaDataReader, ParquetStatisticsPolicy, RowGroupMetaData,
};
use crate::file::reader::{ChunkReader, SerializedPageReader};
use crate::schema::types::SchemaDescriptor;
Expand Down Expand Up @@ -593,8 +593,10 @@ pub struct ArrowReaderOptions {
/// [ARROW_SCHEMA_META_KEY]: crate::arrow::ARROW_SCHEMA_META_KEY
supplied_schema: Option<SchemaRef>,

pub(crate) column_index: PageIndexPolicy,
pub(crate) offset_index: PageIndexPolicy,
Comment on lines -596 to -597

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive by change. Accessors have been added for these, so they no longer need to be visible.

column_index: PageIndexPolicy,
offset_index: PageIndexPolicy,
column_index_mask: ColumnChunkMask,
offset_index_mask: ColumnChunkMask,

/// Options to control reading of Parquet metadata
metadata_options: ParquetMetaDataOptions,
Expand Down Expand Up @@ -774,6 +776,49 @@ impl ArrowReaderOptions {
self
}

// TODO: deprecate the following as part of https://github.com/apache/arrow-rs/issues/11395

/// Sets the same [`ColumnChunkMask`] for both page-index structures.
pub fn with_page_index_mask(self, mask: ColumnChunkMask) -> Self {
self.with_column_index_mask(mask.clone())
.with_offset_index_mask(mask)
}

/// Sets the [`ColumnChunkMask`] for the Parquet [ColumnIndex] structure.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a nice API 👍

///
/// The column index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when filtering by a predicate
/// on a single column). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// The mask applies only if the column-index policy is not [`PageIndexPolicy::Skip`]
/// (the default), or an underlying reader is configured to preload the index. It is
/// honored by loading APIs such as [`ArrowReaderMetadata::load`];
/// [`ArrowReaderMetadata::try_new`] does not load or filter page indexes.
///
/// [ColumnIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interaction between PageIndexPolicy and the ColumnChunkMask is complicated, especially when there is an existing PageIndex in ParquetMetadata

One potential API we could consider is a new variant in PageIndexPolicy::Mask to avoid having to thread through another set of variables.

pub enum PageIndexPolicy {
    /// Do not read the page index.
    #[default]
    Skip,
...
    /// Ensure the specified indexes exist, and error if not
    Required(ColumnChunkMask),
    /// Load the specified indexes if they exist
    Optional(ColumnChunkMask),
}

However, that would be an API change as PageIndexPolicy isn't marked as non_exhastive so we would have to either

  1. Wait for the next major release
  2. Go with this API, and then unify PageIndexPolicy, and deprecate with_column_index_mask et al methods in next major version 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My original design was something similar, but instead added three new variants to the policy which allowed for column only selection, row only selection, or both for a grid. The ColumnChunkMask grew out of that as a way to get the behavior without a breaking API change. I didn't want to code that up, but LLMs make the plumbing changes too easy 😅.

Anyway, I agree that's a cleaner solution that requires less added baggage. I guess the question is how soon do we want this? Given the work @mkleen is doing with direct parsing of the column index in #11285, maybe there's less urgency here and we can wait for the breaking window to open for 61.0.0.

I'm not in love with option 2.

Tangent: since the change to optional cells, I've wondered if we still need the Required vs Optional distinction any longer. The policy enum was introduced when offset indexes were all or nothing, and we needed to decide what to do if a file lacked some, but not all, cells since there was no way to have a placeholder for a missing cell. Now that each cell is optional, the offset index behavior can mirror the column index. If we're waiting for a breaking change, perhaps we can deprecate Required and only support Optional 🤷

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tangent: since the change to optional cells, I've wondered if we still need the Required vs Optional distinction any longer. The policy enum was introduced when offset indexes were all or nothing, and we needed to decide what to do if a file lacked some, but not all, cells since there was no way to have a placeholder for a missing cell. Now that each cell is optional, the offset index behavior can mirror the column index. If we're waiting for a breaking change, perhaps we can deprecate Required and only support Optional 🤷

I think at least at some point in the past the Required variant was to fail fast when a user didn't correctly configure / provide the PageIndex to the parquet reader and the caller wanted to ensure they were properly using the index. Maybe the APIs are sophisticated enough these days that we could just add a flag to ArrowReaderOptions 🤔 . I agree the idea of Required for the metadata reader doesn't really make a lot of sense.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The origin story is that back in the day, the column index had a NONE placeholder variant, so a missing column index was replaced with that. The offset index had no such placeholder, as the belief was all offset indexes would be populated (which makes sense, you can't do page pruning without them). The offset index parser simply panic'd if an individual cell was missing. To get rid of the panic, the policy was introduced; if the policy was Required, then an error was returned if an index was missing, if Optional then the entire offset index would be thrown away and decoding stopped. Now that individual cells of the offset index can be None, there's less need for Required. I think we should at least change the policy to Optional when the old with_page_index(true) is used.

pub fn with_column_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.column_index_mask = mask;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it worth making this API return Result and checking that the column index mask doesn't refer to out of bounds columns (e.g. some column index greater than the number of columns?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that's a good point. I think the PageIndexProvider interface suffers from being infallible (which I'm trying to address in #11159).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked at this, but I think I'll leave it as-is. No other setters here return a Result, so it would be kind of weird. Also, we won't have the footer yet, so we can't really know the proper bounds. Also, it's a mask...nothing wrong with mask bits set that will always be 0 after an AND. It's probably more important to do bounds checking where the mask is used.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also modified the mask api to require bounds when querying mask membership. e17b71f

self
}

/// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure.
///
/// The offset index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when projecting a small subset
/// of columns). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// Page pruning also needs the offset index for predicate columns, so callers should
/// include those columns in addition to projected columns. The same loading and policy
/// qualifications as [`Self::with_column_index_mask`] apply.
///
/// [OffsetIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
pub fn with_offset_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.offset_index_mask = mask;
self
}
Comment on lines +787 to +820

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs / Bug (C3). These docs suggest a column index mask for the predicate columns and an offset index mask for the projected columns. With that setup, StatisticsConverter returns arrays of different lengths (see the review body):

CI columns([0]) (predicate), OI columns([1]) (projection): data_page_mins len 30, data_page_row_counts len 0

Page pruning needs the offset index of the predicate columns too. The suggestion also adds two other traps (C7): a mask without a policy does nothing, and ArrowReaderMetadata::try_new ignores the masks.

Suggested change
/// Sets the [`ColumnChunkMask`] for the Parquet [ColumnIndex] structure.
///
/// The column index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when filtering by a predicate
/// on a single column). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// [ColumnIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
pub fn with_column_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.column_index_mask = mask;
self
}
/// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure.
///
/// The offset index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when projecting a small subset
/// of columns). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// [OffsetIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
pub fn with_offset_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.offset_index_mask = mask;
self
}
/// Sets the [`ColumnChunkMask`] for the Parquet [ColumnIndex] structure.
///
/// The column index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when filtering by a predicate
/// on a single column). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// The mask applies only if the column index policy is not [`PageIndexPolicy::Skip`]
/// (the default), and only when the page index is loaded with these options (for
/// example by [`ArrowReaderMetadata::load`]). [`ArrowReaderMetadata::try_new`] ignores it.
///
/// [ColumnIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
pub fn with_column_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.column_index_mask = mask;
self
}
/// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure.
///
/// The offset index can be costly to decode and store, especially when it is needed
/// only for a subset of row groups or columns (such as when projecting a small subset
/// of columns). Providing a [`ColumnChunkMask`] can greatly decrease
/// the time needed to decode this metadata.
///
/// Page pruning with the column index also needs the offset index of the same
/// columns. So include the predicate columns in this mask, not only the projected
/// columns. The same notes as for [`Self::with_column_index_mask`] apply.
///
/// [OffsetIndex]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
pub fn with_offset_index_mask(mut self, mask: ColumnChunkMask) -> Self {
self.offset_index_mask = mask;
self
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one's getting a little pedantic 😅. I mean, I think users can figure out that a function that explicitly states that it won't load the page index won't be affected by options controlling the loading of the page index.

Fair point re noting that you'll want the offset index for the predicate columns as well as the projected columns.


/// Provide a Parquet schema to use when decoding the metadata. The schema in the Parquet
/// footer will be skipped.
///
Expand Down Expand Up @@ -924,6 +969,22 @@ impl ArrowReaderOptions {
self.column_index
}

// TODO: deprecate the following as part of https://github.com/apache/arrow-rs/issues/11395

/// Retrieve the currently set [`ColumnChunkMask`] for the offset index.
///
/// This can be set via [`with_offset_index_mask`][Self::with_offset_index_mask].
pub fn offset_index_mask(&self) -> &ColumnChunkMask {
&self.offset_index_mask
}

/// Retrieve the currently set [`ColumnChunkMask`] for the column index.
///
/// This can be set via [`with_column_index_mask`][Self::with_column_index_mask].
pub fn column_index_mask(&self) -> &ColumnChunkMask {
&self.column_index_mask
}

/// Retrieve the currently set metadata decoding options.
pub fn metadata_options(&self) -> &ParquetMetaDataOptions {
&self.metadata_options
Expand Down Expand Up @@ -973,6 +1034,18 @@ impl ParquetMetaDataReader {
.with_offset_index_policy(options.offset_index_policy());
}

// Preload settings on the underlying reader may enable an index even when the
// corresponding options policy is `Skip`, so apply non-default masks independently.
//
// TODO: https://github.com/apache/arrow-rs/issues/11395 may deprecate this API. Decide
// at that time if masks should be added to ParquetObjectReader.
if !options.column_index_mask().is_all() {
self = self.with_column_index_mask(options.column_index_mask().clone());
}
if !options.offset_index_mask().is_all() {
self = self.with_offset_index_mask(options.offset_index_mask().clone());
}

self
}
}
Expand Down Expand Up @@ -1019,6 +1092,8 @@ impl ArrowReaderMetadata {
let metadata = ParquetMetaDataReader::new()
.with_column_index_policy(options.column_index_policy())
.with_offset_index_policy(options.offset_index_policy())
.with_column_index_mask(options.column_index_mask().clone())
.with_offset_index_mask(options.offset_index_mask().clone())
.with_metadata_options(Some(options.metadata_options.clone()));
#[cfg(feature = "encryption")]
let metadata = metadata.with_decryption_properties(
Expand Down
9 changes: 9 additions & 0 deletions parquet/src/arrow/arrow_reader/statistics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2042,6 +2042,15 @@ impl<'a> StatisticsConverter<'a> {
let mut nulls = NullBufferBuilder::new(0);
for rg_idx in row_group_indices {
let Some(offset_index) = page_index.offset_index(*rg_idx, parquet_index) else {
// A partial page index may contain the ColumnIndex but not the OffsetIndex.
// Preserve page-wise alignment with data_page_{mins,maxes,null_counts} by
// emitting one null row count for every page known from the ColumnIndex.
let num_pages = page_index
.column_index(*rg_idx, parquet_index)
.map(|index| index.num_pages() as usize)
.unwrap_or(0);
row_counts.resize(row_counts.len() + num_pages, 0);
nulls.append_n_nulls(num_pages);
continue;
};
let page_locations = offset_index.page_locations();
Expand Down
10 changes: 10 additions & 0 deletions parquet/src/arrow/async_reader/store.rs
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,16 @@ impl AsyncFileReader for ParquetObjectReader {
.with_column_index_policy(options.column_index_policy())
.with_offset_index_policy(options.offset_index_policy());
}
// TODO: https://github.com/apache/arrow-rs/issues/11395 may deprecate this API. Decide
// at that time if masks should be added to ParquetObjectReader.
if let Some(options) = options {
if !options.column_index_mask().is_all() {
metadata = metadata.with_column_index_mask(options.column_index_mask().clone());
}
if !options.offset_index_mask().is_all() {
metadata = metadata.with_offset_index_mask(options.offset_index_mask().clone());
}
}

let metadata = if let Some(file_size) = self.file_size {
metadata.load_and_finish(self, file_size).await?
Expand Down
2 changes: 1 addition & 1 deletion parquet/src/file/metadata/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ use crate::{
pub use footer_tail::FooterTail;
pub use options::{ParquetMetaDataOptions, ParquetStatisticsPolicy};
pub use push_decoder::ParquetMetaDataPushDecoder;
pub use reader::{PageIndexPolicy, ParquetMetaDataReader};
pub use reader::{ColumnChunkMask, PageIndexPolicy, ParquetMetaDataReader};
use std::io::Write;
use std::ops::Range;
use std::sync::Arc;
Expand Down
Loading
Loading