Repository navigation
Conversation
|
run benchmark metadata baseline:
ref: b682ebecff792b5d2efdd455633a4b7492335010
env:
BENCH_FILTER: page index |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing page_index_grid (02b5c1d) to b682ebe diff Run configurationrun benchmark metadata
env:
BENCH_FILTER: "page index"
baseline:
ref: "b682ebecff792b5d2efdd455633a4b7492335010"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadata File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing page_index_grid (02b5c1d) to b682ebe diff Run configurationrun benchmark metadata
env:
BENCH_FILTER: "page index"
baseline:
ref: "b682ebecff792b5d2efdd455633a4b7492335010"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
@etseidl do we have benchmarks for wide tables where we do dense/full reads vs. sparse reads along the different pruning directions? |
Reads of the data, or just reads of the page index? |
Presumably both would be valuable, but I'm not sure how much we do full e2e data read benchmarks here vs. datafusion. Even if we just had the page index ones (which we could easily merge as a precursor PR) then it would strengthen the story here. I asked only because I saw #11159 (comment) was basically flat but I assume it's not measuring the thing we are tryign to improve here anwya. |
Ahh, thanks. Answering in reverse order, yeah, that bench (added in #11157) is just demonstrating that selectively reading the page index is much faster than reading the whole thing. The run you referenced is just showing that using the Grid didn't negatively impact reading the thing. We don't have a whole lot of end-to-end benchmarks in this crate, and I'm pretty sure the ones we do have don't do any pruning. I agree it would be nice to have something to demonstrate the value here. With this particular PR, I think the value comes more from saving memory without adversely affecting performance. To that end, we probably should add some kind of a bench that does range based reads with column projection to see if the metadata handling enters into the equation in a meaningful way. I'm sure at huge scales it will. Maybe I'll have Codex/Claude whip up a test for me 😅 |
|
@adriangb I had Codex create a simple point lookup test for me so I could do some benchmarking at varying table widths. The code generates a table with a specified number of rows and columns, then does a series of random point lookups filtering on a "key" column that is unique and sorted. The queries return a varying number of columns (the "projected" column below). It's naive, so each query opens the file, parses the footer metadata, and then optionally parses the page index (both a full scan a la what's done currently, and a read that's targeted to just the predicate and projected columns). Pruning is either not done at all, done at the row group level using the chunk metadata, or done at the page level using the index. (Note this is against a file on NVMe, fully in buffer cache, so it's really just measuring the decoding work. Files in object store would change the calculus immensely :P). At 100 columns, things look pretty ok, but even at this size reading the full page index slows things down relative to just using the column stats. The page index pruning clearly helps here. At 1000 columns you begin to see metadata times dominating the query time. The selective page index is still beating row group pruning, but reading the entire index is a big L. This continues at the 10000 column size. For a single projected column, even the filtered page index barely loses to simply using column stats, but by 25 columns it squeaks ahead. Again, reading the full page index by this point is worse than not pruning at all! So I think the advantage to pruning the page index is pretty apparent here. The need for being able to selectively populate the file metadata is also pretty clear. In the short term we could try the skip index route, but I'm cautiously optimistic we might get a redo of the footer metadata done right. A side effect of this study was seeing how difficult it was to get decent query performance out of the existing APIs. Codex originally went through the ArrowReaderMetadata API, but that at first resulted in re-decoding the footer metadata, and later cloning it. I switched to using the ParquetMetaDataReader where I could re-use the footer metadata directly when reading the page index and that helped immensely. |
|
One last interesting tidbit. I tried running the benchmark with the |
| self | ||
| } | ||
|
|
||
| /// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure. |
There was a problem hiding this comment.
Should it be called ChunkMask then instead of ColumnChunkMask?
There was a problem hiding this comment.
I did leave a note about this in #11157 (#11157 (comment)), but I suppose either could be correct, and ChunkMask is even shorter 😄. I opted for the current name because "column chunk" seems to be the most common name for the set of pages for a given column in a given row group. Happy to revisit this if adding Column is superfluous.
| let num_row_groups = metadata.num_row_groups(); | ||
| let num_columns = metadata.file_metadata().schema_descr().num_columns(); | ||
| let mut builder = PageIndexBuilder::default(); | ||
| let mut builder = PageIndexBuilder::new_with_mask( |
There was a problem hiding this comment.
Won't this preallocate stuff on skip?
There was a problem hiding this comment.
Yep 😅. Thanks. An earlier rev passed in the policy as well, but that got lost in translation. I'll fix that.
c68118d to
7053422
Compare
| { | ||
| *column_slot = Some(column_index); | ||
| } | ||
| ) -> bool { |
There was a problem hiding this comment.
breaking change, this should be reverted (and same for put_offset_index)
There was a problem hiding this comment.
reverted...but still need a fallible version
|
closing this in favor of #11475 |
Which issue does this PR close?
Rationale for this change
The current
PageIndexuses a nested vector of vectors to hold the page indexes. This wastes a great deal of memory when only a few columns or row groups are desired.What changes are included in this PR?
Following a suggestion from @adriangb, this replaces the dense vector of vectors with a new
Gridstructure. Testing has shown that this structure can greatly reduce memory usage, while incurring an acceptable performance hit on index lookup.Are these changes tested?
Yes, new tests are added
Are there any user-facing changes?
No, only internal storage structures are changed
AI assistance
The original idea is from Claude Code, and I had Codex refine it a bit. My own madness is sprinkled in as well.