You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][Parquet] Reject outlandish values in DELTA_BINARY_PACKED decoder #50314
In DeltaBitPackDecoder::InitHeader we're allocating scratch space for miniblock decoding. The size of this scratch space is entirely driven by values read from the DELTA_BINARY_PACKED header, and could therefore be outlandishly huge.
Given that a huge miniblock size is unlikely to be seen in the wild (and would defeat the point of having miniblocks), we should probably reject non-reasonable values in InitHeader.
How do we choose the threshold for non-reasonable values? Maybe we can determine an upper bound from trusted information like the buffer size, rather than have to pick an arbitrary threshold. Eg. looking at the Rust implementation, they don't resize mini_block_bit_widths until reading them here and then throw an error if they fail to read the expected number of bit widths.
How do we choose the threshold for non-reasonable values?
A conservative guesstimate is good enough IMHO. The underlying design of the encoding should suggest that a miniblock larger than 1024 values, for example, does not really make sense.
Maybe we can determine an upper bound from trusted information like the buffer size, rather than have to pick an arbitrary threshold.
The problem is that a range of equal values can encode to exactly 0 bits.
Eg. looking at the Rust implementation, they don't resize mini_block_bit_widths until reading them here
We could mirror that in C++, though that's slightly more logic to write than simply disallowing extremely unlikely values.
Activity
pitrou commentedon Jul 1, 2026
cc @wgtmac @adamreeve @rok
adamreeve commentedon Jul 5, 2026
How do we choose the threshold for non-reasonable values? Maybe we can determine an upper bound from trusted information like the buffer size, rather than have to pick an arbitrary threshold. Eg. looking at the Rust implementation, they don't resize
mini_block_bit_widthsuntil reading them here and then throw an error if they fail to read the expected number of bit widths.pitrou commentedon Jul 6, 2026
A conservative guesstimate is good enough IMHO. The underlying design of the encoding should suggest that a miniblock larger than 1024 values, for example, does not really make sense.
The problem is that a range of equal values can encode to exactly 0 bits.
We could mirror that in C++, though that's slightly more logic to write than simply disallowing extremely unlikely values.
apacheGH-50314: [C++][Parquet] Limit DELTA_BINARY_PACKED miniblock count
apacheGH-50314: [C++][Parquet] Limit DELTA_BINARY_PACKED miniblock count
apacheGH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED hea…
apacheGH-50314: [C++][Parquet] Avoid single-value miniblock allocation
apacheGH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED hea…
apacheGH-50314: [C++][Parquet] Avoid single-value miniblock allocation
apacheGH-50314: [C++][Parquet] Size miniblock buffer after min delta
apacheGH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED hea…
apacheGH-50314: [C++][Parquet] Avoid single-value miniblock allocation
apacheGH-50314: [C++][Parquet] Size miniblock buffer after min delta
apacheGH-50314: Validate DELTA_BINARY_PACKED bit-width bounds
apacheGH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED hea…
apacheGH-50314: [C++][Parquet] Avoid single-value miniblock allocation
apacheGH-50314: [C++][Parquet] Size miniblock buffer after min delta
apacheGH-50314: Validate DELTA_BINARY_PACKED bit-width bounds
GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers (#…
pitrou commentedon Nov 16, 2026
Issue resolved by pull request 51128
#51128
pitrou commentedon Nov 16, 2026
@1fanwang Can you post here so that we assign this issue to you?
1fanwang commentedon Nov 16, 2026
posting, thanks @pitrou