Skip to content

GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers - #51128

Merged
pitrou merged 5 commits into
apache:mainfrom
1fanwang:1fannnw/reject-invalid-delta-headers
Nov 16, 2026
Merged

pitrou merged 5 commits into
apache:mainfrom
1fanwang:1fannnw/reject-invalid-delta-headers

Conversation

@1fanwang

@1fanwang 1fanwang commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

A corrupt Parquet page can trigger a large scratch allocation from its header, even when it contains only one value.

Fixes #50314.

What changes are included in this PR?

Before allocating, the decoder checks that the remaining input has at least one bit-width byte per required miniblock. Block decoding stays separate from this header check. Single-value pages skip the unused buffer.

Are these changes tested?

Yes, by existing and new unit tests.

Are there any user-facing changes?

Valid pages decode as before. Headers exceeding the input bound fail before allocation; truncated block data still produces the decoder's EOF error.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Copilot AI lite review requested due to automatic review settings September 1, 2026 22:32
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The decoder change still allocates miniblock scratch space unconditionally (including for single-value pages) and the new guard should account for required min_delta_ bytes, leaving a remaining allocation-DoS gap that should be closed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the C++ Parquet DELTA_BINARY_PACKED decoder against corrupt page headers that can otherwise drive disproportionate memory allocations and produce misleading EOF errors, aligning behavior with the security/robustness goals described in GH-50314.

Changes:

  • Add header validation in DeltaBitPackDecoder::InitHeader to reject pages whose miniblock count is incompatible with the remaining input bytes and emit a more actionable ParquetException.
  • Add new encoding tests to cover the single-value page path and the corrupt-header rejection case (including allocation behavior via ProxyMemoryPool).
File summaries
File Description
cpp/src/parquet/decoder.cc Adds early validation/error reporting for invalid DELTA_BINARY_PACKED miniblock headers (and aims to prevent oversized allocations).
cpp/src/parquet/encoding_test.cc Adds regression tests for single-value decoding and for rejecting invalid miniblock-width headers without allocating.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cpp/src/parquet/decoder.cc Resolved
Comment thread cpp/src/parquet/encoding_test.cc Resolved
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot AI review requested due to automatic review settings September 3, 2026 19:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes directly address the allocation-before-validation issue with clear guards and are covered by focused regression tests.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot AI review requested due to automatic review settings September 22, 2026 02:24
@1fanwang
1fanwang requested a review from HuaHuaY as a code owner September 22, 2026 02:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Header validation can still allow allocation when min_delta_ uses multiple bytes and width data is truncated.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 24, 2026 06:31
@1fanwang
1fanwang force-pushed the 1fannnw/reject-invalid-delta-headers branch from bb561d0 to 18ae360 Compare September 24, 2026 06:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

Remaining feedback is limited to non-blocking test-name nits.

Review effort: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 25, 2026 20:52
@1fanwang
1fanwang force-pushed the 1fannnw/reject-invalid-delta-headers branch from 18ae360 to fca97c6 Compare September 25, 2026 20:52
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes are covered by regression tests, with only a minor diagnostic wording nit remaining.

Review effort: Lite
Findings: None

@pitrou

pitrou commented Oct 6, 2026

Copy link
Copy Markdown
Member

Sorry for the delay here. I don't like the proposed approach because it starts decoding a block in InitHeader. This makes the code more complicated and brittle.

I think we can do simpler. We know that each miniblock requires at least one byte for its bitwidth.
So the check can look like:

    const int64_t required_blocks = CeilDiv(total_value_count_ - 1, values_per_block_);
    if (required_blocks * mini_blocks_per_block > decoder_->bytes_left()) { /* throw exception */ }

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot AI lite review requested due to automatic review settings October 6, 2026 18:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified.

Review effort: Lite
Findings: None

@1fanwang

Copy link
Copy Markdown
Contributor Author

We know that each miniblock requires at least one byte for its bitwidth.

@pitrou thanks, that bound is simpler than decoding a block. InitHeader now throws when the required bit-width bytes exceed the bytes left, before any allocation. Done in 025e748, with tests for truncated headers.

Copilot AI lite review requested due to automatic review settings November 10, 2026 14:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

Review effort: Lite

Comment thread cpp/src/parquet/decoder.cc Resolved
Comment on lines +1940 to +1950
EXPECT_THROW_THAT(
[&] {
decoder->SetData(2, encoded.data(), static_cast<int>(encoded.size()));
decoder->Decode(decoded.data(), static_cast<int>(decoded.size()));
},
ParquetException,
::testing::Property(
&ParquetException::what,
::testing::HasSubstr("the required number of miniblock bit widths "
"(1048576) exceeds the number of bytes remaining "
"in the data (1)")));
Comment on lines +1948 to +1950
::testing::HasSubstr("the required number of miniblock bit widths "
"(1048576) exceeds the number of bytes remaining "
"in the data (1)")));
1fanwang and others added 5 commits November 16, 2026 14:46
…ders

InitHeader() sizes the bit-width buffer from the header's miniblock
count without tying it to the page size, so a 10-byte page claiming
2^20 miniblocks allocates 1 MiB before failing. InitBlock() reads one
bit-width byte per miniblock, so such a page can never decode.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Single-value pages never initialize a block, so leave the bit-width buffer unallocated. Account for the required min-delta byte when validating block metadata, and use cumulative allocation counts in the regression tests.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
InitHeader subtracted one byte for min delta, so a two-byte min delta
with no bit widths still allocated the aligned scratch buffer and then
failed with Decode bit-width EOF. Consume min delta first, then reject
when remaining bytes cannot hold the declared miniblock widths.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@pitrou
pitrou force-pushed the 1fannnw/reject-invalid-delta-headers branch from 2515222 to 2ca5762 Compare November 16, 2026 14:10
Copilot AI lite review requested due to automatic review settings November 16, 2026 14:10

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thanks for the update @1fanwang . I'm going watch for CI and merge if nothing regressed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
1 resolved since last review

Review effort: Lite

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

@pitrou
pitrou merged commit 5580366 into apache:main Nov 16, 2026
53 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Reject outlandish values in DELTA_BINARY_PACKED decoder

4 participants