Skip to content

Encode and decode SimpleAggregateFunction alias chains below Nullable - #32

Merged
joe-clickhouse merged 2 commits into
mainfrom
joe/nested-saf-validate
Nov 18, 2026
Merged

joe-clickhouse merged 2 commits into
mainfrom
joe/nested-saf-validate

Conversation

@joe-clickhouse

@joe-clickhouse joe-clickhouse commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Inserting SimpleAggregateFunction(f, Nullable(T)) inside Array, Tuple, Map, Nested or Variant failed with a mismatched column error. Top-level columns worked.

column_variant_matches only stripped Nullable at the top level. The container arms called .inner() before resolving the alias, so the child reached the matcher as a Nullable with no matching arm. The matcher now strips Nullable at every level.

The encoder, decoder, skip path and resumable stream check also resolved only one alias level below Nullable. So Nullable(SAF(f, SAF(g, T))) or Nullable(SAF(f, Point)) could not be written or read. All four now use resolved_physical_delegate_ref.

Type names in the header and the nullability checks are unchanged.


Note

Medium Risk
Touches core native encode/decode/skip paths and batch validation for nullable and alias types; behavior change is localized but affects all nested SAF+Nullable wire handling.

Overview
Fixes encode_block rejecting columns typed as SimpleAggregateFunction(..., Nullable(T)) (including chained SAF aliases) when nested in Array, Tuple, Map, Nested, or Variant, which previously surfaced as a mismatched-column / InconsistentBatch error while top-level columns worked.

column_variant_matches now unwraps Nullable at every recursion level and matches container children against the full type (no extra .inner() stripping before alias resolution), so physical column buffers align with nested nullable SAF shapes.

Encode, decode, skip, and resumable stream paths switch from single-level physical_delegate_ref to resolved_physical_delegate_ref below Nullable, so chains like Nullable(SAF(..., SAF(...))) or Nullable(SAF(..., Point)) round-trip. Type headers and nullability rules are unchanged.

Adds broad roundtrip, validation, rev0 byte-framing, and StreamDecoder coverage for these types, and updates a stream scanner test that now treats the framed block as complete instead of UnsupportedType.

Reviewed by Cursor Bugbot for commit 8f8bfa4. Bugbot is set up for automated code reviews on this repo. Configure here.

column_variant_matches stripped Nullable only at the top level and
container arms called inner() before resolving aliases, so
SimpleAggregateFunction(f, Nullable(T)) inside Array, Tuple, Map, Nested
and Variant was rejected as a mismatched column. The encoder, decoder,
skip path and resumable stream check also resolved only one alias level
below Nullable.
@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 18, 2026 18:28

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on 61101e0. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

security-ai-gate-reviewer Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

AI review gate — auto-approved

8f8bfa4b2859 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • 8f8bfa4b2859 approved — green: Cursor Bugbot
  • 61101e0ea45e approved — green: Cursor Bugbot

polyglotAI-bot
polyglotAI-bot previously approved these changes Nov 18, 2026

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the fix against the code and against a live server.

  • The root cause matches the description. The container arms in column_variant_matches peeled .inner() before alias resolution, and the four wire paths expanded only one alias level below Nullable. All four now resolve the full chain, the same as empty_column already did.
  • No Nullable, Array, or LowCardinality can appear after the resolve. can_be_inside_nullable in the parser also resolves the full chain, and encode re-parses the rendered header.
  • State prefix and suffix, is_encodable, and the Arrow export already recurse through aliases at each level, so no other site needs the change.
  • No hot-path cost. The resolve runs once per column, and for a non-alias inner it is the same single match as before.

Checks on 61101e0:

  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, and cargo +1.81 check --workspace --all-targets pass.
  • With the three production files reverted to main, 5 of the new or changed tests fail. With the PR code they pass.
  • Live, server 26.10.4.207: I used the 14 type shapes from the new tests (Array, Tuple, named Tuple, Map, Nested, Variant, chained SAF, SAF over Point) with null maps [0, 1, 0] and [1, 1]. The server accepts each type in CREATE TABLE, the crate's INSERT ... FORMAT Native succeeds, and the server's SELECT ... FORMAT Native bytes are byte-identical to the bytes the crate sent. The decoded values match, and the Arrow export of each decoded batch succeeds.

One non-blocking note inline, about an error-class change for a caller-built Nullable(Nullable(T)).

Optional follow-up: the live suite has no column of this shape yet. A column such as Array(SimpleAggregateFunction(max, Nullable(UInt64))) in geo_saf_nested_roundtrip_through_server would pin the reported INSERT case against a real server.

Comment thread src/native/encode/validate.rs Resolved
@github-actions

Copy link
Copy Markdown

AI review gate — Human approval in place of AI review

Approved on the head commit by chernser (write access, not the author), standing in for the AI review:

  • Cursor Bugbot is still reviewing this commit.

What to do:

  • Nothing to do yet if the review is running. The gate turns green when it finishes clean.
  • No review showing up? Post bugbot run as a new top-level PR comment. If that does nothing, Bugbot is probably not enabled for this repository: ask a repository administrator.

Evaluated at 8f8bfa4b. This required workflow does not re-run on its own once the AI review later concludes — check the Cursor Bugbot check on this commit directly for its current state.

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on 8f8bfa4. Details in the approver's comment on this pull request.

@joe-clickhouse
joe-clickhouse merged commit 8f2e4b3 into main Nov 18, 2026
6 checks passed
@joe-clickhouse
joe-clickhouse deleted the joe/nested-saf-validate branch November 18, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants