Skip to content

Use Grid storage for Parquet PageIndex - #11159

Closed
etseidl wants to merge 96 commits into
apache:mainfrom
etseidl:page_index_grid
Closed

etseidl wants to merge 96 commits into
apache:mainfrom
etseidl:page_index_grid

Conversation

@etseidl

@etseidl etseidl commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The current PageIndex uses a nested vector of vectors to hold the page indexes. This wastes a great deal of memory when only a few columns or row groups are desired.

What changes are included in this PR?

Following a suggestion from @adriangb, this replaces the dense vector of vectors with a new Grid structure. Testing has shown that this structure can greatly reduce memory usage, while incurring an acceptable performance hit on index lookup.

Are these changes tested?

Yes, new tests are added

Are there any user-facing changes?

No, only internal storage structures are changed

AI assistance

The original idea is from Claude Code, and I had Codex refine it a bit. My own madness is sprinkled in as well.

@etseidl etseidl added the parquet Changes to the parquet crate label Sep 21, 2026
@etseidl

etseidl commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

This is stacked on #11157. Diff against that branch is here.

@etseidl

etseidl commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

run benchmark metadata

baseline:
  ref: b682ebecff792b5d2efdd455633a4b7492335010
env:
  BENCH_FILTER: page index

@adriangbot

Copy link
Copy Markdown

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5767041854-2561-6w7hv 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing page_index_grid (02b5c1d) to b682ebe diff

Run configuration
run benchmark metadata
env:
  BENCH_FILTER: "page index"
baseline:
  ref: "b682ebecff792b5d2efdd455633a4b7492335010"

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadata
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing page_index_grid (02b5c1d) to b682ebe diff

Run configuration
run benchmark metadata
env:
  BENCH_FILTER: "page index"
baseline:
  ref: "b682ebecff792b5d2efdd455633a4b7492335010"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                              main                                   page_index_grid
-----                              ----                                   ---------------
open(page index)                   1.01    131.5±0.38µs        ? ?/sec    1.00    130.8±0.42µs        ? ?/sec
read page index                    1.00    123.1±0.11µs        ? ?/sec    1.00    122.7±0.12µs        ? ?/sec
read page index reduced columns    1.00     13.3±0.01µs        ? ?/sec    1.01     13.4±0.01µs        ? ?/sec

Resource Usage

base (merge-base)

Metric Value
Wall time 35.0s
Peak memory 11.2 MiB
Avg memory 7.0 MiB
CPU user 27.8s
CPU sys 0.0s
Peak spill 0 B

branch

Metric Value
Wall time 30.0s
Peak memory 83.0 MiB
Avg memory 10.8 MiB
CPU user 28.0s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@adriangb

Copy link
Copy Markdown
Contributor

@etseidl do we have benchmarks for wide tables where we do dense/full reads vs. sparse reads along the different pruning directions?

@etseidl

etseidl commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

do we have benchmarks for wide tables where we do dense/full reads vs. sparse reads along the different pruning directions?

Reads of the data, or just reads of the page index?

@adriangb

Copy link
Copy Markdown
Contributor

Reads of the data, or just reads of the page index?

Presumably both would be valuable, but I'm not sure how much we do full e2e data read benchmarks here vs. datafusion. Even if we just had the page index ones (which we could easily merge as a precursor PR) then it would strengthen the story here. I asked only because I saw #11159 (comment) was basically flat but I assume it's not measuring the thing we are tryign to improve here anwya.

@etseidl

etseidl commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Reads of the data, or just reads of the page index?

Presumably both would be valuable, but I'm not sure how much we do full e2e data read benchmarks here vs. datafusion. Even if we just had the page index ones (which we could easily merge as a precursor PR) then it would strengthen the story here. I asked only because I saw #11159 (comment) was basically flat but I assume it's not measuring the thing we are tryign to improve here anwya.

Ahh, thanks. Answering in reverse order, yeah, that bench (added in #11157) is just demonstrating that selectively reading the page index is much faster than reading the whole thing. The run you referenced is just showing that using the Grid didn't negatively impact reading the thing.

We don't have a whole lot of end-to-end benchmarks in this crate, and I'm pretty sure the ones we do have don't do any pruning. I agree it would be nice to have something to demonstrate the value here. With this particular PR, I think the value comes more from saving memory without adversely affecting performance. To that end, we probably should add some kind of a bench that does range based reads with column projection to see if the metadata handling enters into the equation in a meaningful way. I'm sure at huge scales it will.

Maybe I'll have Codex/Claude whip up a test for me 😅

@etseidl

etseidl commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

@adriangb I had Codex create a simple point lookup test for me so I could do some benchmarking at varying table widths. The code generates a table with a specified number of rows and columns, then does a series of random point lookups filtering on a "key" column that is unique and sorted. The queries return a varying number of columns (the "projected" column below). It's naive, so each query opens the file, parses the footer metadata, and then optionally parses the page index (both a full scan a la what's done currently, and a read that's targeted to just the predicate and projected columns). Pruning is either not done at all, done at the row group level using the chunk metadata, or done at the page level using the index. (Note this is against a file on NVMe, fully in buffer cache, so it's really just measuring the decoding work. Files in object store would change the calculus immensely :P).

At 100 columns, things look pretty ok, but even at this size reading the full page index slows things down relative to just using the column stats. The page index pruning clearly helps here.

projected  pruning                 metadata size   footer/query     index load     index eval     data/query    total/query
        1  none                       456.30 KiB      687.412µs            0ns            0ns    13.695408ms    14.451238ms
        1  row group                  456.30 KiB      599.734µs            0ns            0ns     1.454493ms     2.117207ms
        1  page index (full)         1012.61 KiB      588.467µs     1.200928ms          920ns      601.318µs     2.456797ms
        1  page index (selective)     457.08 KiB      595.609µs       91.832µs          842ns      608.451µs     1.359397ms
       25  none                       456.30 KiB      661.783µs            0ns            0ns    40.989044ms     41.72175ms
       25  row group                  456.30 KiB      676.718µs            0ns            0ns     8.570402ms     9.316573ms
       25  page index (full)         1012.61 KiB      633.917µs     1.372732ms        1.118µs     6.442088ms     8.519533ms
       25  page index (selective)     461.50 KiB      677.392µs       115.79µs        1.003µs     5.989914ms     6.851483ms

At 1000 columns you begin to see metadata times dominating the query time. The selective page index is still beating row group pruning, but reading the entire index is a big L.

projected  pruning                 metadata size   footer/query     index load     index eval     data/query    total/query
        1  none                         4.45 MiB     6.487408ms            0ns            0ns     15.08447ms    21.897408ms
        1  row group                    4.45 MiB      6.50841ms            0ns            0ns     1.841498ms      8.66429ms
        1  page index (full)            9.88 MiB     5.747513ms    16.112672ms        1.421µs     1.200488ms    23.402837ms
        1  page index (selective)       4.45 MiB     5.611439ms      275.741µs        1.095µs      730.992µs       6.9187ms
       25  none                         4.45 MiB     6.461029ms            0ns            0ns    45.832282ms     52.61971ms
       25  row group                    4.45 MiB     7.006866ms            0ns            0ns    10.332351ms    17.681206ms
       25  page index (full)            9.88 MiB     6.549746ms    17.171582ms        1.476µs     7.764038ms    31.846389ms
       25  page index (selective)       4.45 MiB     6.285695ms      348.461µs        1.229µs     6.565722ms    13.510559ms

This continues at the 10000 column size. For a single projected column, even the filtered page index barely loses to simply using column stats, but by 25 columns it squeaks ahead. Again, reading the full page index by this point is worse than not pruning at all!

projected  pruning                 metadata size   footer/query     index load     index eval     data/query    total/query
        1  none                        44.46 MiB     61.37101ms            0ns            0ns    16.463509ms    80.517332ms
        1  row group                   44.46 MiB     57.46164ms            0ns            0ns     2.734904ms    62.666304ms
        1  page index (full)           98.77 MiB    55.465729ms   158.227488ms        1.914µs      1.78146ms    218.15268ms
        1  page index (selective)      44.46 MiB    59.214822ms     1.969541ms        1.702µs      1.79331ms    65.421946ms
       25  none                        44.46 MiB     67.72911ms            0ns            0ns    48.087823ms   118.719795ms
       25  row group                   44.46 MiB    67.420542ms            0ns            0ns    12.070592ms    82.314726ms
       25  page index (full)           98.77 MiB    63.277779ms   175.549233ms        2.056µs      9.84537ms   251.711592ms
       25  page index (selective)      44.46 MiB    63.131708ms     2.451171ms        1.934µs     8.800725ms    77.007979ms

So I think the advantage to pruning the page index is pretty apparent here. The need for being able to selectively populate the file metadata is also pretty clear. In the short term we could try the skip index route, but I'm cautiously optimistic we might get a redo of the footer metadata done right.

A side effect of this study was seeing how difficult it was to get decent query performance out of the existing APIs. Codex originally went through the ArrowReaderMetadata API, but that at first resulted in re-decoding the footer metadata, and later cloning it. I switched to using the ParquetMetaDataReader where I could re-use the footer metadata directly when reading the page index and that helped immensely.

@etseidl

etseidl commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

One last interesting tidbit. I tried running the benchmark with the Vec<Vec<Option<T>>> version of the page index and was getting very strange behavior. The time to parse the footer went up for cases where the page index was in use. Turns out that creating and dropping all those nested vecs was trashing memory. Another reason to go with the Grid. Thanks for suggesting it @adriangb 😄

self
}

/// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be called ChunkMask then instead of ColumnChunkMask?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did leave a note about this in #11157 (#11157 (comment)), but I suppose either could be correct, and ChunkMask is even shorter 😄. I opted for the current name because "column chunk" seems to be the most common name for the set of pages for a given column in a given row group. Happy to revisit this if adding Column is superfluous.

Comment thread parquet/src/file/metadata/parser.rs Outdated
let num_row_groups = metadata.num_row_groups();
let num_columns = metadata.file_metadata().schema_descr().num_columns();
let mut builder = PageIndexBuilder::default();
let mut builder = PageIndexBuilder::new_with_mask(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this preallocate stuff on skip?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep 😅. Thanks. An earlier rev passed in the policy as well, but that got lost in translation. I'll fix that.

{
*column_slot = Some(column_index);
}
) -> bool {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

breaking change, this should be reverted (and same for put_offset_index)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted...but still need a fallible version

@etseidl

etseidl commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

closing this in favor of #11475

@etseidl etseidl closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support load page index metadata with specified column id instead of all columns

5 participants