Repository navigation
GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers - #51128
Conversation
|
|
There was a problem hiding this comment.
🟡 Changes recommended
The decoder change still allocates miniblock scratch space unconditionally (including for single-value pages) and the new guard should account for required min_delta_ bytes, leaving a remaining allocation-DoS gap that should be closed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the C++ Parquet DELTA_BINARY_PACKED decoder against corrupt page headers that can otherwise drive disproportionate memory allocations and produce misleading EOF errors, aligning behavior with the security/robustness goals described in GH-50314.
Changes:
- Add header validation in
DeltaBitPackDecoder::InitHeaderto reject pages whose miniblock count is incompatible with the remaining input bytes and emit a more actionableParquetException. - Add new encoding tests to cover the single-value page path and the corrupt-header rejection case (including allocation behavior via
ProxyMemoryPool).
File summaries
| File | Description |
|---|---|
| cpp/src/parquet/decoder.cc | Adds early validation/error reporting for invalid DELTA_BINARY_PACKED miniblock headers (and aims to prevent oversized allocations). |
| cpp/src/parquet/encoding_test.cc | Adds regression tests for single-value decoding and for rejecting invalid miniblock-width headers without allocating. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
There was a problem hiding this comment.
🟢 Approval recommended
The changes directly address the allocation-before-validation issue with clear guards and are covered by focused regression tests.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
|
bb561d0 to
18ae360
Compare
18ae360 to
fca97c6
Compare
|
|
|
Sorry for the delay here. I don't like the proposed approach because it starts decoding a block in I think we can do simpler. We know that each miniblock requires at least one byte for its bitwidth. const int64_t required_blocks = CeilDiv(total_value_count_ - 1, values_per_block_);
if (required_blocks * mini_blocks_per_block > decoder_->bytes_left()) { /* throw exception */ } |
|
|
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open issues
Review effort: Lite
| EXPECT_THROW_THAT( | ||
| [&] { | ||
| decoder->SetData(2, encoded.data(), static_cast<int>(encoded.size())); | ||
| decoder->Decode(decoded.data(), static_cast<int>(decoded.size())); | ||
| }, | ||
| ParquetException, | ||
| ::testing::Property( | ||
| &ParquetException::what, | ||
| ::testing::HasSubstr("the required number of miniblock bit widths " | ||
| "(1048576) exceeds the number of bytes remaining " | ||
| "in the data (1)"))); |
| ::testing::HasSubstr("the required number of miniblock bit widths " | ||
| "(1048576) exceeds the number of bytes remaining " | ||
| "in the data (1)"))); |
…ders InitHeader() sizes the bit-width buffer from the header's miniblock count without tying it to the page size, so a 10-byte page claiming 2^20 miniblocks allocates 1 MiB before failing. InitBlock() reads one bit-width byte per miniblock, so such a page can never decode. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Single-value pages never initialize a block, so leave the bit-width buffer unallocated. Account for the required min-delta byte when validating block metadata, and use cumulative allocation counts in the regression tests. Signed-off-by: 1fanwang <1fannnw@gmail.com>
InitHeader subtracted one byte for min delta, so a two-byte min delta with no bit widths still allocated the aligned scratch buffer and then failed with Decode bit-width EOF. Consume min delta first, then reject when remaining bytes cannot hold the declared miniblock widths. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
2515222 to
2ca5762
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
2 open issues
1 resolved since last review
Review effort: Lite
|
|



Rationale for this change
A corrupt Parquet page can trigger a large scratch allocation from its header, even when it contains only one value.
Fixes #50314.
What changes are included in this PR?
Before allocating, the decoder checks that the remaining input has at least one bit-width byte per required miniblock. Block decoding stays separate from this header check. Single-value pages skip the unused buffer.
Are these changes tested?
Yes, by existing and new unit tests.
Are there any user-facing changes?
Valid pages decode as before. Headers exceeding the input bound fail before allocation; truncated block data still produces the decoder's EOF error.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: