Skip to content

GH-55669: [C++][IPC] Validate buffer bounds and replace raw pointer arithmetic in MessageDecoder - #55671

Open
Awesome06 wants to merge 5 commits into
apache:mainfrom
Awesome06:gh-55669-ipc-buffer-bounds
Open

Awesome06 wants to merge 5 commits into
apache:mainfrom
Awesome06:gh-55669-ipc-buffer-bounds

Conversation

@Awesome06

Copy link
Copy Markdown
Contributor

Rationale for this change

In arrow::ipc::MessageDecoderImpl, chunk reassembly relied on unchecked pointer arithmetic (memcpy(out + offset, ...)) during streaming message reconstruction. Furthermore, MessageDecoder::Consume(data, size) did not validate that size >= 0, which could construct invalid negative-sized Buffer instances and corrupt internal chunk accounting. In addition, FlatBuffers tensor metadata decoders called .size() directly on tensor->shape() without verifying that the optional vector was non-null, and IPC readers lacked opt-in limits to constrain untrusted metadata sizes, body lengths, or decompression expansion ratios.

What changes are included in this PR?

  • Replaces raw pointer arithmetic in MessageDecoderImpl::ConsumeDataChunks with std::span and checked addition via ::arrow::internal::AddWithOverflow.
  • Adds DCHECK_GE invariants on chunk sizing in ConsumeDataChunks.
  • Enforces validation in MessageDecoder::Consume rejecting size < 0 at the public API entry point.
  • Adds CHECK_FLATBUFFERS_NOT_NULL guards for tensor->shape(), dimension elements, and custom metadata pairs in metadata_internal.cc.
  • Adds max_metadata_length, max_body_length, and max_decompression_ratio to IpcReadOptions (defaulting to -1 for unlimited / full backwards compatibility).
  • Enforces max_decompression_ratio in DecompressBuffer using MultiplyWithOverflow and slices compressed payloads with std::span.
  • Plumbs IpcReadOptions through MessageDecoder constructors and StreamDecoderImpl.

Are these changes tested?

Yes. Existing IPC streaming and file tests pass. New deterministic unit tests were added in cpp/src/arrow/ipc/read_write_test.cc:

  • TestMessageDecoderBounds.NegativeConsumeSize
  • TestMessageDecoderBounds.NegativeContinuationToken
  • TestMessageDecoderBounds.NegativeMetadataLength
  • TestMessageDecoderBounds.MaxMetadataLengthExceeded
  • TestStreamDecoderLimits.MaxBodyLengthExceeded
  • TestDecompressBufferLimits.NegativeUncompressedSize
  • TestDecompressBufferLimits.ZeroCompressedPayloadWithNonZeroUncompressed
  • TestDecompressBufferLimits.MaxDecompressionRatioExceeded
  • TestMetadataBounds.EmptyOrInvalidTensorMetadata

Are there any user-facing changes?

No breaking changes. Users can optionally configure max_metadata_length, max_body_length, and max_decompression_ratio on IpcReadOptions when reading IPC streams.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

…in MessageDecoder

* Replace raw pointer arithmetic and unchecked memcpy offsets in MessageDecoderImpl with std::span and AddWithOverflow

* Enforce validation against negative data size in MessageDecoder::Consume

* Add opt-in max_metadata_length, max_body_length, and max_decompression_ratio ceilings to IpcReadOptions (defaulting to unlimited: -1)

* Use CHECK_FLATBUFFERS_NOT_NULL guards in Tensor metadata decoders

* Add deterministic unit tests covering negative sizes, opt-in limits, and decompression bounds in read_write_test
Copilot AI lite review requested due to automatic review settings November 18, 2026 13:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change still has an unresolved critical gap where configured IPC size limits are ignored on file-reader paths, plus a decompression-ratio boundary bug that allows unlimited expansion when set to zero.

2 open issues
What changed in this PR

This PR hardens C++ IPC decoding against malformed or oversized input by validating message lengths, protecting chunk reassembly, checking FlatBuffers fields, and adding configurable decompression and allocation limits.

Changes:

  • Adds bounds validation for decoder input, metadata, body lengths, and chunk copying.
  • Adds null checks for tensor and custom metadata FlatBuffers fields.
  • Adds configurable metadata, body, and decompression-ratio limits to IpcReadOptions, with associated tests.
File Description
cpp/​src/​arrow/​ipc/​reader.cc Adds decompression-ratio validation and propagates read options to streaming decoding.
cpp/​src/​arrow/​ipc/​reader_internal.h Exposes the internal decompression helper declaration.
cpp/​src/​arrow/​ipc/​read_write_test.cc Adds tests for decoder bounds, metadata validation, body limits, and decompression limits.
cpp/​src/​arrow/​ipc/​options.h Adds configurable IPC metadata, body, and decompression limits.
cpp/​src/​arrow/​ipc/​metadata_internal.cc Validates required FlatBuffers tensor and custom metadata fields.
cpp/​src/​arrow/​ipc/​message.h Adds MessageDecoder constructors accepting IpcReadOptions.
cpp/​src/​arrow/​ipc/​message.cc Validates input sizes and message lengths, uses spans for chunk copying, and enforces decoder limits.

Review effort: Lite

Comment thread cpp/src/arrow/ipc/options.h Resolved
Comment thread cpp/src/arrow/ipc/reader.cc Resolved
…r and enforce reader limits

* Avoid use-after-free in MessageDecoderImpl when listener OnEOS resets the decoder
* Early exit when data span is fully consumed before trailing member dereferences
* Wire IpcReadOptions through ReadMessage, MessageReader, and RecordBatchFileReader
* Add block bounds validation and test coverage for IPC reader limits
Copilot AI lite review requested due to automatic review settings November 18, 2026 16:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval is blocked by two active moderate issues: invalid negative length limits can disable IPC protections, and file-footer metadata is not covered by the configured metadata bound.

3 open issues
1 resolved since last review

Review effort: Lite

Comment thread cpp/src/arrow/ipc/options.h Resolved
Comment thread cpp/src/arrow/ipc/reader.cc Resolved
…force file footer bound

* Add IpcReadOptions::Validate() to reject negative values below -1
* Wire options validation to stream and file reader entry points
* Enforce max_metadata_length against footer_length in ReadFooterAsync
* Add tests for footer bounds, negative options validation, and decoder rejection
Copilot AI lite review requested due to automatic review settings November 18, 2026 18:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The supplied MessageReader path bypasses the new metadata and body size limits, leaving custom readers such as Flight’s vulnerable to unbounded message decoding.

2 open issues
1 resolved since last review

Review effort: Lite

… MessageReader streams

* Validate metadata and body lengths in StreamDecoderInternal::OnMessageDecoded
* Protect custom MessageReader pathways against unbounded allocations
* Add unit test coverage with MockCustomMessageReader
Copilot AI lite review requested due to automatic review settings November 18, 2026 19:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the change is unusually complex and a critical hidden issue indicates untrusted input can trigger oversized prebuffer reads before configured limits are enforced.

2 open issues

Review effort: Lite

Copilot AI lite review requested due to automatic review settings November 18, 2026 19:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because moderate and critical findings indicate the hardening may still permit allocation denial-of-service and allow the advertised body-length limit to be bypassed on a public reader path.

No open issues 🎉

2 resolved since last review

Review effort: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants