Skip to content

[C++][Parquet] Reject outlandish values in DELTA_BINARY_PACKED decoder #50314

Closed

Description

@pitrou

Describe the enhancement requested

In DeltaBitPackDecoder::InitHeader we're allocating scratch space for miniblock decoding. The size of this scratch space is entirely driven by values read from the DELTA_BINARY_PACKED header, and could therefore be outlandishly huge.

Given that a huge miniblock size is unlikely to be seen in the wild (and would defeat the point of having miniblocks), we should probably reject non-reasonable values in InitHeader.

Component(s)

C++, Parquet

Activity

  1. pitrou commented on Jul 1, 2026

    @pitrou
    MemberAuthor
  2. adamreeve commented on Jul 5, 2026

    @adamreeve
    Contributor

    How do we choose the threshold for non-reasonable values? Maybe we can determine an upper bound from trusted information like the buffer size, rather than have to pick an arbitrary threshold. Eg. looking at the Rust implementation, they don't resize mini_block_bit_widths until reading them here and then throw an error if they fail to read the expected number of bit widths.

  3. pitrou commented on Jul 6, 2026

    @pitrou
    MemberAuthor

    How do we choose the threshold for non-reasonable values?

    A conservative guesstimate is good enough IMHO. The underlying design of the encoding should suggest that a miniblock larger than 1024 values, for example, does not really make sense.

    Maybe we can determine an upper bound from trusted information like the buffer size, rather than have to pick an arbitrary threshold.

    The problem is that a range of equal values can encode to exactly 0 bits.

    Eg. looking at the Rust implementation, they don't resize mini_block_bit_widths until reading them here

    We could mirror that in C++, though that's slightly more logic to write than simply disallowing extremely unlikely values.

  4. added 4 commits that reference this issue on Sep 1, 2026
  5. added 6 commits that reference this issue on Sep 24, 2026
  6. added a commit that references this issue on Oct 6, 2026
  7. added 5 commits that reference this issue on Nov 16, 2026
  8. added this to the 27.0.0 milestone on Nov 16, 2026
  9. pitrou commented on Nov 16, 2026

    @pitrou
    MemberAuthor

    Issue resolved by pull request 51128
    #51128

  10. pitrou commented on Nov 16, 2026

    @pitrou
    MemberAuthor

    @1fanwang Can you post here so that we assign this issue to you?

  11. 1fanwang commented on Nov 16, 2026

    @1fanwang
    Contributor

    @1fanwang Can you post here so that we assign this issue to you?

    posting, thanks @pitrou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions