Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
1c77984
allow for serialization of custom page index providers
etseidl Sep 8, 2026
b2d3a1a
clippy
etseidl Sep 8, 2026
0638b87
add test of using custom page index provider with row selection
etseidl Sep 9, 2026
da1ff75
refactor and add usage stats
etseidl Sep 18, 2026
60e4b4c
Merge branch 'main' into page_index_serialization
etseidl Sep 21, 2026
a39dd8f
add test of using custom page index provider with row selection
etseidl Sep 9, 2026
30631fa
refactor and add usage stats
etseidl Sep 18, 2026
5458721
allow for serialization of custom page index providers
etseidl Sep 8, 2026
979141a
clippy
etseidl Sep 8, 2026
3a8ee6a
add plumbing for page index selective population
etseidl Sep 19, 2026
1df722e
more tests
etseidl Sep 19, 2026
9eacc74
rename PageIndexSelection to ColumnChunkMask
etseidl Sep 20, 2026
69ab61a
docs and cleanup
etseidl Sep 20, 2026
cf9d819
Merge branch 'main' into page_index_roundtrip
etseidl Sep 21, 2026
b7b529d
more miri friendly
etseidl Sep 21, 2026
b682ebe
rework tests to do no file i/o
etseidl Sep 21, 2026
bd587b2
apply suggestions from review
etseidl Sep 23, 2026
6a6e62f
Merge remote-tracking branch 'origin/main' into page_index_serialization
etseidl Sep 23, 2026
c5350f5
Merge branch 'page_index_serialization' of github.com:etseidl/arrow-r…
etseidl Sep 23, 2026
997f184
remove assert
etseidl Sep 23, 2026
0969c23
apply suggestion from review
etseidl Sep 23, 2026
a3d54e1
Merge branch 'main' into page_index_roundtrip
etseidl Sep 23, 2026
d93796e
add another assert
etseidl Sep 23, 2026
2a2fd09
clippy
etseidl Sep 23, 2026
c687e6e
restore old page index location behavior
etseidl Sep 23, 2026
55eb967
fix docs
etseidl Sep 23, 2026
92fb20c
honor preserve_page_index_locations in write_x_indexes
etseidl Sep 23, 2026
e63ad47
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 23, 2026
e31d45b
Merge branch 'page_index_serialization' into page_index_selection
etseidl Sep 23, 2026
2489741
implement review suggestions
etseidl Sep 23, 2026
8351703
clippy
etseidl Sep 23, 2026
76ec6bf
Merge remote-tracking branch 'origin/main' into page_index_roundtrip
etseidl Sep 23, 2026
57d7030
tests now pass
etseidl Sep 23, 2026
ac9a08e
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 23, 2026
49e6c76
Merge branch 'page_index_roundtrip' into page_index_selection
etseidl Sep 23, 2026
82087c5
apply suggestion from review
etseidl Sep 23, 2026
7b30fa0
revert most changes to push decoder
etseidl Sep 24, 2026
3cefc59
address a number of issues found in review, primarily documentation
etseidl Sep 24, 2026
4ee4125
modify range_for_page_index to take masks
etseidl Sep 24, 2026
f6a8fde
more efficient page index parser traversal
etseidl Sep 24, 2026
682edbf
address StatisticsConverter misalignment
etseidl Sep 24, 2026
18f34a8
enable C3 test
etseidl Sep 24, 2026
1ec36bb
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 24, 2026
8169311
fix C11: preserve existing indexes across multiple parse calls
etseidl Sep 24, 2026
09337c8
Fix page index loading with suffix prefetch
etseidl Sep 24, 2026
5b9c07d
add plumbing for page index selective population
etseidl Sep 19, 2026
1c30c6a
more tests
etseidl Sep 19, 2026
3177084
rename PageIndexSelection to ColumnChunkMask
etseidl Sep 20, 2026
942f2bf
docs and cleanup
etseidl Sep 20, 2026
199a41d
rework tests to do no file i/o
etseidl Sep 21, 2026
3dfeeca
apply suggestion from review
etseidl Sep 23, 2026
2696260
revert most changes to push decoder
etseidl Sep 24, 2026
e303c53
address a number of issues found in review, primarily documentation
etseidl Sep 24, 2026
9672e90
modify range_for_page_index to take masks
etseidl Sep 24, 2026
1c5a135
more efficient page index parser traversal
etseidl Sep 24, 2026
2c597fb
address StatisticsConverter misalignment
etseidl Sep 24, 2026
134a3b9
enable C3 test
etseidl Sep 24, 2026
dc23d99
fix C11: preserve existing indexes across multiple parse calls
etseidl Sep 24, 2026
78917ed
empty_index no longer needs pub(crate)
etseidl Sep 25, 2026
fe9af28
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
64a34bf
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
063a084
Merge branch 'page_index_selection' of github.com:etseidl/arrow-rs in…
etseidl Sep 25, 2026
b99f1bf
Merge remote-tracking branch 'origin/main' into page_index_selection
etseidl Sep 25, 2026
9fd6d8a
Revert "fix C11: preserve existing indexes across multiple parse calls"
etseidl Sep 25, 2026
bc975c4
replace path for C11
etseidl Sep 25, 2026
0a9dab9
restore previous compatibility exception for PageIndexPolicy::Required
etseidl Sep 25, 2026
372d065
remove mention of the C11 tag
etseidl Sep 25, 2026
ec46a88
switch to grid storage for page index
etseidl Sep 21, 2026
f56a82d
rework Keep to use i32 internally
etseidl Sep 21, 2026
c11421e
revert change to i32
etseidl Sep 24, 2026
c2844e6
remove PageIndexBuilder::new_with_mask and signal when put_X fails
etseidl Sep 25, 2026
25dca4c
fix heap_size
etseidl Sep 25, 2026
7053422
make new fallible
etseidl Sep 25, 2026
5c7719b
fix some merge problems
etseidl Sep 25, 2026
9b87ed6
D5: partialeq for grid
etseidl Sep 26, 2026
455b342
D6: fix Grid::from_vec
etseidl Sep 26, 2026
3b7a078
D10: doc fixes
etseidl Sep 26, 2026
8e8494e
add ability to create Keep from a chunk mask
etseidl Sep 26, 2026
8334423
add tests
etseidl Sep 26, 2026
e1e3201
revert breaking api changes
etseidl Sep 26, 2026
9dc6483
check that index insertion succeeds
etseidl Sep 26, 2026
0accf0d
document possible double counting in PageIndex::heap_size
etseidl Sep 26, 2026
81af27b
revert changes to the put_x_index fns
etseidl Sep 26, 2026
3f421e0
clippy
etseidl Sep 26, 2026
220c6c4
add reshape APIs to the PageIndex and document
etseidl Sep 28, 2026
c217708
remove linear search for small spans
etseidl Sep 28, 2026
19f7125
add fallible put functions to PageIndexBuilder
etseidl Sep 28, 2026
3ab2fd4
finish docs for parse_page_index
etseidl Sep 28, 2026
c47bb75
import ProjectionMask
etseidl Sep 28, 2026
b9fd884
address nit
etseidl Sep 29, 2026
ad86518
replace boxed iterator with more targeted concrete iterator impl
etseidl Sep 29, 2026
4823984
revert to using single passed in Bytes
etseidl Sep 29, 2026
1242c66
Merge branch 'page_index_selection' into page_index_grid
etseidl Sep 30, 2026
2d67e59
Merge remote-tracking branch 'origin/main' into page_index_grid
etseidl Sep 30, 2026
f6be8f0
Merge remote-tracking branch 'origin/main' into page_index_grid
etseidl Oct 7, 2026
7ff415c
remove functions re-introduced during merge
etseidl Oct 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions parquet/src/arrow/async_reader/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1008,7 +1008,7 @@ mod tests {
use crate::basic::Encoding;
use crate::file::metadata::PageIndexPolicy;
use crate::file::metadata::ParquetMetaDataReader;
use crate::file::metadata::page_index::PageIndex;
use crate::file::metadata::page_index::{Grid, PageIndex};
use crate::file::properties::WriterProperties;
use arrow::compute::kernels::cmp::eq;
use arrow::error::Result as ArrowResult;
Expand Down Expand Up @@ -1970,7 +1970,7 @@ mod tests {
.await
.unwrap();

let page_index = PageIndex::new(None, Some(vec![]));
let page_index = PageIndex::new(None, Some(Grid::from_vec(vec![]).unwrap()));
metadata.set_page_index(Some(Arc::new(page_index)));
let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
let arrow_reader_metadata = ArrowReaderMetadata::try_new(metadata.into(), options).unwrap();
Expand Down
4 changes: 2 additions & 2 deletions parquet/src/file/metadata/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2122,9 +2122,9 @@ mod tests {

// Size with page index (includes Arc overhead plus PageIndex heap size)
#[cfg(not(feature = "encryption"))]
let bigger_expected_size = 3280;
let bigger_expected_size = 3328;
#[cfg(feature = "encryption")]
let bigger_expected_size = 3448;
let bigger_expected_size = 3496;

// more set fields means more memory usage
assert!(bigger_expected_size > base_expected_size);
Expand Down
Loading
Loading