Repository navigation
Append zero-width array runs in constant time - #30
Conversation
decode_binary_value and encode_binary_value looped once per element for arrays and maps of empty tuples, which take no bytes on the wire. Runs of zero-width values now append in one step per tuple node. Decode also rejects FixedString(0) and QBit(T, 0), as encode already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Auto-approved by the AI review gate on dcd5b9f. Details in the approver's comment on this pull request.
AI review gate — auto-approved
History (latest 5 heads, newest first)
|
6efe1e1
There was a problem hiding this comment.
Auto-approved by the AI review gate on 6efe1e1. Details in the approver's comment on this pull request.
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Reviewed at 6efe1e1. The merge from main does not change the diff of this PR.
What I checked:
- fmt, clippy -D warnings,
cargo test --workspaceand the 1.81 MSRV check pass. - The fix works. Release build, main -> this PR:
Arrayof a 97-deepTuple()with a count of 1M, 3 input bytes: decode 247 ms -> 0.13 ms, encode 275 ms -> 2 us.Array(Array(Tuple())), 100 x 1M: decode 196 ms -> 2 us, encode 235 ms -> 1.4 us.
zero_width_runs_take_constant_stepsfails when I turn off the decode fill or only the encode skip. So the step counters cover both paths.- The encode skip is safe. For a row below the bound, the per-row path does only checks that the bound already covers: field count, tuple validity, row < len, and the LowCardinality dictionary. The probe follows the same alias arms as
encode_value. So the first failing row and its error stay the same. fillalways comes with the 1M cap: each builder thatis_zero_widthaccepts also passeshas_zero_width_values.zero_width_dynamic_rev0.nativeis byte-identical when I capture it again from server 26.8.39.19.- Plain types:
Array(UInt8)x 1M shows no change. Many small plain arrays encode a bit slower:Array(Array(UInt8))with 500k runs of one, 4.0 ms -> 4.5 ms.#[inline]onzero_width_bounddoes not change it. I think this is OK for the single-value path.
Two notes inline. Neither blocks the merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e5d2dfa
|
Thanks for measuring the small-array encode cost. About 1 ns per run on the single-value path seems fine to me, so I'm leaving it. |
There was a problem hiding this comment.
Auto-approved by the AI review gate on e5d2dfa. Details in the approver's comment on this pull request.
decode_binary_valueandencode_binary_valuelooped once per element for arrays and maps of empty tuples. Those take no bytes on the wire, so a tiny value with a large count and a nested type could cost seconds of CPU.FixedString(0)andQBit(T, 0), as encode already did. The type parsers never produce them.The per-container limit of 1M elements is unchanged.
Note
Medium Risk
Touches core binary value encode/decode paths for nested containers; behavior change is bounded by existing caps but affects error timing and invalid-type handling.
Overview
Fixes a DoS-style CPU blow-up in native single-value helpers when
Array/Mapcounts point at zero-byteTuple()(or nested all-empty tuples): decode/encode no longer iterate once per counted element and instead bulk-append or skip encoding per tuple node, while keeping the 1M per-container cap.Decode now rejects caller-built
FixedString(0)andQBit(T, 0)asInvalid, aligned with encode. Encode still validates child column lengths and switches to the per-row path when tuple validity, LowCardinality dictionaries, or shape mismatches require it, so errors stay the same as before.Adds a live Native fixture for Dynamic SharedVariant cells with large zero-width array counts, plus visit-count tests and integration round-trip coverage.
Reviewed by Cursor Bugbot for commit e5d2dfa. Bugbot is set up for automated code reviews on this repo. Configure here.