Skip to content

Append zero-width array runs in constant time - #30

Merged
joe-clickhouse merged 3 commits into
mainfrom
joe/zero-width-array-budget
Nov 19, 2026
Merged

joe-clickhouse merged 3 commits into
mainfrom
joe/zero-width-array-budget

Conversation

@joe-clickhouse

@joe-clickhouse joe-clickhouse commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

decode_binary_value and encode_binary_value looped once per element for arrays and maps of empty tuples. Those take no bytes on the wire, so a tiny value with a large count and a nested type could cost seconds of CPU.

  • Runs of zero-width values now append in one step per tuple node.
  • Encode checks the run once, then falls back to the per-row path at the first bad row. Errors match main.
  • Decode rejects FixedString(0) and QBit(T, 0), as encode already did. The type parsers never produce them.

The per-container limit of 1M elements is unchanged.


Note

Medium Risk
Touches core binary value encode/decode paths for nested containers; behavior change is bounded by existing caps but affects error timing and invalid-type handling.

Overview
Fixes a DoS-style CPU blow-up in native single-value helpers when Array/Map counts point at zero-byte Tuple() (or nested all-empty tuples): decode/encode no longer iterate once per counted element and instead bulk-append or skip encoding per tuple node, while keeping the 1M per-container cap.

Decode now rejects caller-built FixedString(0) and QBit(T, 0) as Invalid, aligned with encode. Encode still validates child column lengths and switches to the per-row path when tuple validity, LowCardinality dictionaries, or shape mismatches require it, so errors stay the same as before.

Adds a live Native fixture for Dynamic SharedVariant cells with large zero-width array counts, plus visit-count tests and integration round-trip coverage.

Reviewed by Cursor Bugbot for commit e5d2dfa. Bugbot is set up for automated code reviews on this repo. Configure here.

decode_binary_value and encode_binary_value looped once per element for
arrays and maps of empty tuples, which take no bytes on the wire. Runs
of zero-width values now append in one step per tuple node. Decode also
rejects FixedString(0) and QBit(T, 0), as encode already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 18, 2026 00:00

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on dcd5b9f. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

security-ai-gate-reviewer Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

AI review gate — auto-approved

e5d2dfadaaea is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • e5d2dfadaaea approved — green: Cursor Bugbot
  • 6efe1e181322 approved — green: Cursor Bugbot
  • dcd5b9f31ec0 approved — green: Cursor Bugbot

chernser
chernser previously approved these changes Nov 18, 2026

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on 6efe1e1. Details in the approver's comment on this pull request.

polyglotAI-bot
polyglotAI-bot previously approved these changes Nov 19, 2026

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 6efe1e1. The merge from main does not change the diff of this PR.

What I checked:

  • fmt, clippy -D warnings, cargo test --workspace and the 1.81 MSRV check pass.
  • The fix works. Release build, main -> this PR:
    • Array of a 97-deep Tuple() with a count of 1M, 3 input bytes: decode 247 ms -> 0.13 ms, encode 275 ms -> 2 us.
    • Array(Array(Tuple())), 100 x 1M: decode 196 ms -> 2 us, encode 235 ms -> 1.4 us.
  • zero_width_runs_take_constant_steps fails when I turn off the decode fill or only the encode skip. So the step counters cover both paths.
  • The encode skip is safe. For a row below the bound, the per-row path does only checks that the bound already covers: field count, tuple validity, row < len, and the LowCardinality dictionary. The probe follows the same alias arms as encode_value. So the first failing row and its error stay the same.
  • fill always comes with the 1M cap: each builder that is_zero_width accepts also passes has_zero_width_values.
  • zero_width_dynamic_rev0.native is byte-identical when I capture it again from server 26.8.39.19.
  • Plain types: Array(UInt8) x 1M shows no change. Many small plain arrays encode a bit slower: Array(Array(UInt8)) with 500k runs of one, 4.0 ms -> 4.5 ms. #[inline] on zero_width_bound does not change it. I think this is OK for the single-value path.

Two notes inline. Neither blocks the merge.

Comment thread CODEC_CONTRACT.md Resolved
Comment thread src/native/binary_value.rs Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe-clickhouse

Copy link
Copy Markdown
Collaborator Author

Thanks for measuring the small-array encode cost. About 1 ns per run on the single-value path seems fine to me, so I'm leaving it.

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on e5d2dfa. Details in the approver's comment on this pull request.

@joe-clickhouse
joe-clickhouse merged commit e56793c into main Nov 19, 2026
6 checks passed
@joe-clickhouse
joe-clickhouse deleted the joe/zero-width-array-budget branch November 19, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants