Skip to content

Decode JSON values inside Dynamic shared cells - #29

Merged
joe-clickhouse merged 2 commits into
mainfrom
joe/json-in-dynamic-shared-variant
Nov 19, 2026
Merged

joe-clickhouse merged 2 commits into
mainfrom
joe/json-in-dynamic-shared-variant

Conversation

@joe-clickhouse

@joe-clickhouse joe-clickhouse commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

JSON values stored in a Dynamic shared variant came back as raw bytes. decode_binary_value returned Unsupported for JSON, and bindings fall back to bytes when that happens.

  • decode_binary_value and encode_binary_value now handle JSON, including inside Array, Tuple, Map and Nullable.
  • Shared paths come out sorted per row, as the server expects.
  • The binary type descriptor limit goes from 1000 to 2001 nodes, so JSON with 1000 typed paths parses. This is our own limit, not the server's.
  • Bulk decode still keeps shared cells opaque.

Note

Medium Risk
Large new codec surface in binary_value.rs for JSON and nested containers, with new bounds and error semantics that bindings may rely on.

Overview
On-demand decode_binary_value / encode_binary_value now materialize structured JSON (typed path columns plus sorted shared path cells), including JSON nested in Array, Tuple, Map, and Nullable. Bulk decode still leaves Dynamic shared blobs opaque; bindings use the helpers per cell.

Encoding enforces the same rules as decode: typed paths need supported leaf codecs; untyped paths can keep Dynamic/Variant tagged as opaque bytes. Non-null Nothing is Invalid, not Unsupported. Binary descriptor complexity rises from 1000 to 2001 nodes (enough for JSON with 1000 scalar typed paths); depth errors read nesting exceeds 100.

SkipPlan compiles once per descriptor so repeated array/map walks don’t re-clone type trees. Real-server fixtures and integration tests cover JSON in shared cells at revisions 0, 54460, and 54485.

Reviewed by Cursor Bugbot for commit a246301. Bugbot is set up for automated code reviews on this repo. Configure here.

@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 17, 2026 23:47

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ec675b3. Configure here.

Comment thread src/native/binary_value.rs Resolved
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

AI review gate — AI review passed

Clean review from: Cursor Bugbot.

chernser
chernser previously approved these changes Nov 18, 2026
polyglotAI-bot
polyglotAI-bot previously approved these changes Nov 19, 2026

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I found no correctness, safety or wire-contract defect. The inline comments are not blocking.

What I checked at ec675b3:

  • Gate in a clean Rust 1.96 container: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace and cargo +1.81 check --workspace --all-targets all pass.
  • Regression proof: with src/native/binary_value.rs and src/native/type_binary.rs reverted to the merge base, the 7 new fixture tests fail. With the PR code they pass.
  • Server source at v26.8.1.2041-lts: SerializationObject::serializeBinary and deserializeBinary, SerializationDynamic::serializeBinary, ColumnDynamic::serializeValueIntoSharedVariant, SerializationNothing and decodeDataTypeImpl agree with the decoder and with the new CODEC_CONTRACT text. Typed paths have no descriptor, other paths have one, Dynamic NULL is the single 0x00 byte, and the cell layout is the same in V1 and V2. When the server reads a value, it sorts shared paths by raw bytes and rejects duplicates by default, so the sort and the duplicate errors here match it. The Variant skip indexes the global discriminator order, which normalize_variant_alternatives gives.
  • Live server 26.8.1.2041: I decoded and re-encoded all 74 shared cells of json_shared_rev0, json_shared_rev54485 and json_multipart_rev0, and encoded the bulk typed and nested columns with encode_binary_value. I inserted them as Dynamic(max_types=0) shared cells and compared dynamicType(d), toString(d) with the original server cells. The result is the same in a Memory table and after a MergeTree OPTIMIZE FINAL. 18 cells have different bytes, and the server reads them the same. A JSON cell in encoded_shared_cells_insert_through_server would keep this covered in CI.

Merge: CHANGELOG.md conflicts with main after #28. The other files merge without conflict.

Nits: the too_deep() message change is not necessary for JSON support. The module doc still says "confirmed at v26.3". NullDefaultBudget now also meters missing typed paths and zero-width defaults, so a more general name would help. encode_json_value repeats the lookup that DynamicColumn::value_position does.

Comment thread src/native/binary_value.rs Resolved
Comment thread src/native/type_binary.rs Resolved
Comment thread src/native/binary_value.rs Resolved
joe-clickhouse and others added 2 commits November 19, 2026 13:12
decode_binary_value and encode_binary_value now handle JSON, including
JSON nested in Array, Tuple, Map and Nullable. Before, they returned
Unsupported and bindings fell back to raw bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop over-charging null JSON defaults, check dynamic child types on
encode, and list SkipPlan variants explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe-clickhouse
joe-clickhouse force-pushed the joe/json-in-dynamic-shared-variant branch from ec675b3 to a246301 Compare November 19, 2026 21:54

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on a246301. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

Copy link
Copy Markdown

AI review gate — auto-approved

a2463013d104 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • a2463013d104 approved — green: Cursor Bugbot

@joe-clickhouse
joe-clickhouse merged commit 5a552e9 into main Nov 19, 2026
6 checks passed
@joe-clickhouse
joe-clickhouse deleted the joe/json-in-dynamic-shared-variant branch November 19, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants