GH-50421: [C++][Parquet] Add LevelDecoder Skip and Count#50422
Open
AntoinePrv wants to merge 13 commits into
Open
GH-50421: [C++][Parquet] Add LevelDecoder Skip and Count#50422AntoinePrv wants to merge 13 commits into
AntoinePrv wants to merge 13 commits into
Conversation
|
|
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Parquet C++ decoding by moving more “skip/count” functionality into LevelDecoder and underlying RLE/bit-packed utilities, with the goal of simplifying skip logic in column/record readers.
Changes:
- Refactors
LevelDecoderto a singlestd::variant-backed implementation and addsSkip/CountUpTo. - Reworks
TypedColumnReaderImpl::Skipto skip within a page by advancing rep/def levels and skipping only the required physical values. - Adds/updates Arrow RLE/bit-packed decoder internals and tests for
Advance/CountUpTo.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| cpp/src/parquet/column_reader.h | Updates LevelDecoder public API/docs to add Skip and CountUpTo and hide implementation details behind a pimpl. |
| cpp/src/parquet/column_reader.cc | Implements new LevelDecoder internals, introduces SkippableTypedDecoder, and simplifies TypedColumnReaderImpl::Skip. |
| cpp/src/arrow/util/rle_encoding_internal.h | Adds Advance / CountUpTo plumbing, introduces BitPackedDecoder, and refactors run processing into ProcessValues. |
| cpp/src/arrow/util/rle_encoding_test.cc | Adds targeted tests for CountUpTo and Advance behavior. |
Comment on lines
+603
to
+605
| value_bit_width_ = value_bit_width; | ||
| const auto value_count = (static_cast<int64_t>(data_size) * 8) / value_bit_width; | ||
| ARROW_DCHECK_LE(value_count, std::numeric_limits<rle_size_t>::max()); |
Comment on lines
+205
to
+208
| const int num_values = std::min(num_values_remaining_, batch_size); | ||
| const int num_advanced = decoder_->Advance(num_values); | ||
| ARROW_DCHECK_EQ(num_values, num_advanced); | ||
| num_values_remaining_ -= num_advanced; |
Comment on lines
+213
to
+216
| const int num_values = std::min(num_values_remaining_, batch_size); | ||
| const auto result = decoder_->CountUpTo(value, num_values); | ||
| ARROW_DCHECK_EQ(num_values, result.advanced_count); | ||
| num_values_remaining_ -= result.advanced_count; |
Comment on lines
+78
to
+80
| /// Initialize the LevelDecoder state with new data from a V2 page. | ||
| /// | ||
| /// Encoding of the level in the |
| /// Use a scratch buffer to decode values into that buffer. | ||
| /// This was migrated here from a historical implementation. | ||
| /// Ideally all decoders would implement a `Skip` functionality that would at best | ||
| /// avoid decoding, and at worst, decode without intermediarry allocation. |
Comment on lines
+316
to
+320
| /// Advance and count the number of occurrence of a values. | ||
| /// | ||
| /// The count is limited to at most the next `batch_size` items. | ||
| /// @return The matching value count and number of of element that were processed. | ||
| RleCountUpToResult CountUpTo(const RleCountUpToParams<value_type>& p) { |
Comment on lines
+90
to
+93
| /// Advance and count the number of occurrence of a values. | ||
| /// | ||
| /// The count is limited to at most the next `batch_size` items. | ||
| /// @return The matching value count and number of elements that were processed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Adding new methods to the
LevelDecoderto start reducing the complexity inColumnReaderImplBase,TypedColumnReaderImpl, andTypedRecordReader.What changes are included in this PR?
LevelDecodermembers based on a single variantLevelDecoder::SkipLevelDecoder::CountUpToSkipimplementation inTypedColumnReaderImplSkippableTypedDecoderto wrapTypedDecoderwith the remaining usage of thescratch_for_skip_bufferAre these changes tested?
Yes.
Are there any user-facing changes?
No.