Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,14 @@ version; patch releases remain backward compatible.
Array, Tuple, Map, Nested, and Variant with a mismatched-column error. Alias
chains below Nullable now encode and decode in any position. Type headers and
nullability checks are unchanged.
- `decode_binary_value` and `encode_binary_value` now support structured JSON
instead of returning `Unsupported`, including JSON inside arrays, tuples, and
maps. These on-demand helpers preserve typed paths and tagged shared paths.
Bulk Dynamic and JSON shared cells stay opaque. Non-null Nothing values now
return `Invalid` in both helpers.
- Binary type descriptors with 1001 to 2001 nodes now decode. The 2001-node
budget is a crate resource limit. Type-depth errors now say
`nesting exceeds 100` instead of `type nesting too deep`.
- ZSTD compression frames (method byte 0x90) now decode. Before, they returned
`CompressionError::UnsupportedCodec`. A ZSTD frame that claims more output
than its payload can hold fails early with `InvalidUncompressedSize`. Real
Expand Down
93 changes: 89 additions & 4 deletions CODEC_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2483,8 +2483,14 @@ parse and value materialization into the bulk path. On demand,
one-row column, and `write_binary_type_prefix` plus `encode_binary_value` write
one row of a column back as a blob, so a caller can build the SharedVariant
child or JSON shared data that `encode_block` takes. The encode side covers the
same types as the decode side and reproduces the server's value bytes.
Descriptors also match, except SimpleAggregateFunction and AggregateFunction
same types as the decode side and uses the server's single-value layout.
This includes JSON at the root or nested in supported containers. Single-value
JSON decodes into the same structured column model as bulk JSON: declared typed
paths keep their child columns, while all other paths use tagged shared-data
buffers. It does not infer a separate Dynamic child for each discovered path.
JSON path order may differ because the server iterates unordered maps.
Descriptors also match, except JSON path lists are canonicalized and
SimpleAggregateFunction and AggregateFunction
use the custom-name `0x2c` form that the server accepts instead of its native
aggregate descriptor forms. The server writes SimpleAggregateFunction with
`0x2e` and AggregateFunction with `0x25`; AggregateFunction value encoding
Expand Down Expand Up @@ -2551,8 +2557,19 @@ headers and Dynamic tables use `DataTypesBinaryEncoding` descriptors instead.
The public `decode_*_binary_types` and `encode_*_binary_types` APIs select that
out-of-band form while keeping `DecodeOptions` and `EncodeOptions` source
compatible. The descriptor reader is bounded by `MAX_TYPE_DEPTH`, complexity
1000, and list count 1,000,000, and rejects illegal wrappers, reserved tags, and
2001, and list count 1,000,000, and rejects illegal wrappers, reserved tags, and
unsupported descriptor forms before payload decode.
The 2001-node budget is a crate resource policy, not server parity. It covers
a JSON root with 1000 Nullable scalar typed paths, while 1000 typed
Array(Nullable(Int64)) paths require 3001 nodes and remain rejected.
ClickHouse decodes shared-cell descriptors without a complexity limit. Its
Native binary-header setting `input_format_binary_max_type_complexity`
defaults to 1000 and can be raised independently. The server's typed-path
count cap remains 1000. These server limits are confirmed at
`v26.8.1.2041-lts` in
`src/DataTypes/DataTypesBinaryEncoding.cpp::decodeDataTypeImpl`,
`src/DataTypes/DataTypeObject.h::MAX_TYPED_PATHS`, and
`src/Core/FormatFactorySettings.h`.

A zero-row Native block carries only the logical column header. NativeWriter
suppresses the structure word, type table, child prefixes, routing, and bodies.
Expand Down Expand Up @@ -2695,11 +2712,79 @@ bytes; the type-aware per-column row-count guard excludes JSON (and any containe
that bottoms out in it) and bounds it by its own read-before-allocate decode.
SharedVariant and shared-data values are opaque binary blobs, one binary
data-type descriptor followed by one `serializeBinary` payload, the same shape as
a Dynamic SharedVariant cell, kept opaque and never materialized. A zero-row
a Dynamic SharedVariant cell, kept opaque during bulk decoding. The
single-value binary helpers can materialize these cells on demand. A zero-row
Native block writes only the column header (no structure word, path list, or
bodies); `empty_column` builds the declared typed paths with empty children and
empty shared data.

**Single-value binary payload:** JSON inside a shared cell uses a different
layout from the bulk structure above. Its type descriptor starts with `0x30`,
then UInt8 descriptor version `0`, VarUInt `max_dynamic_paths`, UInt8
`max_dynamic_types`, a VarUInt-counted typed-path list of path strings and
recursive type descriptors, a VarUInt-counted skip-path list, and a
VarUInt-counted skip-regexp list. Strings use VarUInt byte lengths.

The value starts with a VarUInt path count. Each entry is a path string followed
by its value. Declared typed paths use their declared serializer without another
type descriptor. Other paths carry a binary type descriptor followed by one
value. The server writes typed paths first, then non-null dynamic paths, then
shared-data paths, with no global ordering guarantee. Typed defaults are
included even when a path was absent in the input document. An empty untyped
object is one zero byte. The value has no bulk structure version word or path
table, and its layout is the same inside V1 and V2 Dynamic shared cells.

`decode_binary_value` preserves typed children and stores other entries in
`StructuredJson` shared buffers, sorted by path within each row. Paths are
compared as raw bytes and duplicates are rejected. Untyped path bytes are
preserved without UTF-8 validation, like bulk shared paths.
`encode_binary_value` accepts structured JSON
from either this helper or bulk decoding. Typed paths require supported
single-value codecs, so a declared Variant, Dynamic, or AggregateFunction typed
path still returns `BinaryValueError::Unsupported`. Untyped paths can retain
Variant and Dynamic payloads as opaque tagged cells. When encoding a
bulk-decoded JSON column, only the selected row's runtime values need codecs.
An unrelated row's unsupported child does not prevent encoding. A selected
runtime value that reaches a Dynamic or Variant leaf still returns
`Unsupported`, including such leaves inside nonempty containers. Empty
containers do not visit their leaves and can encode.
A preexisting shared path
tagged Nothing is retained as the one-byte `0x00` cell. Normal JSON input omits
null untyped paths, but the row serializer copies raw shared entries verbatim.
Nothing has no non-null row payload. The opaque value walker permits it only
as a tagged Dynamic NULL, including inside Array(Dynamic). Arrays and maps
with a Nothing element, key, or value type must be empty. Nullable(Nothing)
must carry a null flag.
Non-null Nothing values return `BinaryValueError::Invalid` in both helpers.
Runtime child descriptors written by the JSON helper use the same 2001-node
complexity limit as the descriptor reader.
Missing typed paths and zero-width default nodes consume units from the
per-value materialization budget, even when they allocate no data bytes.
Opaque values are framed with a plan compiled once per binary descriptor.
Aliases and zero-width children are resolved during compilation, so repeated
Array and Map values do not re-walk or clone their type trees.
The helper preserves serialized paths without applying `SKIP` or `SKIP REGEXP`
again. Normal server output has already applied its input skip rules. This
differs from the server's row deserializer, which filters incoming paths using
byte-prefix SKIP matching and RE2 full-match regexps.
The bulk decoder continues to retain shared cells without parsing each one.
These per-value layouts are
**CONFIRMED** at `v26.8.1.2041-lts` in
`src/DataTypes/Serializations/SerializationObject.cpp`
(`SerializationObject::serializeBinary`, `deserializeBinary`),
`src/DataTypes/Serializations/SerializationDynamic.cpp`
(`SerializationDynamic::serializeBinary`),
`src/DataTypes/Serializations/SerializationNothing.h`
(`SerializationNothing::serializeBinary`, `deserializeBinary`),
`src/Columns/ColumnDynamic.cpp`
(`ColumnDynamic::serializeValueIntoSharedVariant`), and
`src/DataTypes/DataTypesBinaryEncoding.cpp`
(`encodeDataTypeImpl`, `decodeDataTypeImpl`).
Shared-path ordering and row skip rules are confirmed in
`src/Columns/ColumnObject.cpp::findPathLowerBoundInSharedData` and
`src/DataTypes/Serializations/SerializationObject.cpp`
(`deserializeBinary`, `shouldSkipPath`).

**Arrow export:** a structured body exports as an Arrow struct (`+s`) whose
children are, in fixed order: one child per declared typed path (each the path
type's own schema and buffers), one Dense Union child per dynamic path (the same
Expand Down
10 changes: 9 additions & 1 deletion COMPLETENESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ default; the user may override it.
types. This crate does not implement that separate layout yet, so
`DBMS_TCP_PROTOCOL_VERSION` and the framed fixture intentionally remain at
negotiated revision 54485. BlockInfo and Geometry framing are unchanged.
- **JSON shared-cell checkpoint, 2026-11-17:** the single-value binary codec
decodes and encodes JSON, including JSON nested in containers. Typed paths
retain their declared columns, and other paths retain tagged shared cells.
Real-server regressions at `v26.8.1.2041-lts` cover revisions 0, 54460 and
54485. Bulk Dynamic and JSON shared buffers stay opaque.
- **AggregateFunction checkpoint:** decode, encode, streaming, Arrow LargeBinary
export, real-server fixtures, and live INSERT coverage are complete for exact
base `count` with zero or one argument, canonical
Expand Down Expand Up @@ -651,7 +656,10 @@ where the across-release churn lives.
under the type-aware row-count guard. `Nullable(JSON)` is legal (top-level
null map), `LowCardinality(JSON)` illegal, and JSON composes in
Array/Tuple/Map/Variant/Dynamic and as its own typed path (nested), bounded
by `MAX_TYPE_DEPTH`. Arrow export is a struct of the typed paths, the
by `MAX_TYPE_DEPTH`. The single-value binary helpers also decode and
encode JSON inside shared cells, preserving typed children and retaining
other paths as tagged shared values. Arrow export is a struct of the
typed paths, the
result-wide dynamic-path Dense Unions, and a `_shared_data` LargeList of
`{paths utf8, values binary}`, with a Text body exporting as utf8; binary
type descriptor tag 0x30. Legacy `Object('json')` is not registered at the
Expand Down
7 changes: 7 additions & 0 deletions INTEGRATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,13 @@ The physical column buffers are Arrow-shaped:

The core covers the full ClickHouse scalar set plus containers (`Array`, `Tuple`, `Map`, `Nested`), `Variant`, `Dynamic`, `JSON`, wide integers, `LowCardinality(T)`, the geo types, `QBit`, a set of `AggregateFunction`/`SimpleAggregateFunction` states, and `Nullable(T)` over the supported inner types. The "Supported types" list in `README.md` is the current enumeration, and `CODEC_CONTRACT.md` is the per-type wire and buffer contract; this guide does not repeat the list so it cannot drift. Unsupported types fail with `DecodeError::UnsupportedType` on decode (and `EncodeError::UnsupportedType` on encode).

Dynamic shared cells and JSON shared-data values retain their binary type
descriptor and single-value payload. Decode them on demand with
`native::binary_value::read_binary_type_prefix` and `decode_binary_value`.
This supports JSON at the root and inside containers. JSON typed paths retain
their declared child columns, and its other paths use tagged shared-data
buffers. Bindings can process those buffers like standalone JSON.

ClickHouse `String` is arbitrary bytes, not guaranteed UTF-8. The core stores the raw bytes. The current Arrow export uses Arrow utf8 format because the physical layout is offsets plus data, but strict Arrow consumers may reject invalid UTF-8 when importing or later validating the array. A binding that needs byte-faithful behavior should validate first, choose a bytes/binary fallback, or return a clear unsupported-for-Arrow error for invalid strings.

## Consumption Strategies
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,11 @@ block bytes for `INSERT` with `native::encode::encode_block` or
`decode_*_binary_types` entry points. Bytes from `encode_*_binary_types` require
`input_format_native_decode_types_in_binary_format=1` on the receiving INSERT.

Dynamic shared cells and JSON shared-data values stay as binary blobs during
bulk decode. Use `native::binary_value::read_binary_type_prefix` and
`decode_binary_value` to decode a cell on demand, including JSON inside
containers. The corresponding writers build cells for encoding.

## Repo layout

The root package remains `ch-core-rs`. The workspace also contains
Expand Down
55 changes: 55 additions & 0 deletions scripts/gen_fixtures.sh
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,24 @@ FORMAT Native
SQL
)

json_shared_query=$(
cat <<'SQL'
WITH number AS n
SELECT
CAST(multiIf(n = 0, '{"n":{"x":13},"z":79,"a":17}', n = 1, '{}', n = 2, '{"n":{"x":null}}', '{"n":{"x":79}}')::JSON, 'Dynamic(max_types=0)') AS j,
CAST([multiIf(n = 0, '{"k":1}', n = 1, '{}', n = 2, '{"other":null}', '{"k":79}')::JSON, '{}'::JSON], 'Dynamic(max_types=0)') AS aj,
CAST(tuple('{"k":79}'::JSON, toUInt8(17)), 'Dynamic(max_types=0)') AS tj,
CAST(map('user_1', '{"k":13}'::JSON), 'Dynamic(max_types=0)') AS mj,
CAST(multiIf(n = 0, '{"a":13,"b":"user_1","d":79,"dropped":17,"omit_me":1}', n = 1, '{}', n = 2, '{"a":null,"d":null}', '{"a":79,"later":13}') AS JSON(max_dynamic_paths=0, max_dynamic_types=0, a Nullable(Int64), b String, SKIP dropped, SKIP REGEXP '^omit')) AS typed,
CAST(typed, 'Dynamic(max_types=0)') AS typed_shared,
CAST('{"mix":[1,"user_1",null]}'::JSON, 'Dynamic(max_types=0)') AS heterogeneous,
CAST('{"child":{"k":13}}' AS JSON(child JSON)) AS nested,
CAST(nested, 'Dynamic(max_types=0)') AS nested_shared
FROM numbers(4)
FORMAT Native
SQL
)

capture() {
local name="$1"
local query="$2"
Expand Down Expand Up @@ -583,4 +601,41 @@ capture "json_flattened_rev0.native" "${json_aux_query}" \

capture "zero_width_dynamic_rev0.native" "${zero_width_dynamic_query}"

# JSON single-value payloads inside Dynamic SharedVariant cells. Revisions 0
# and 54460 exercise Dynamic V1, while 54485 exercises V2.
json_mixed_rows_query=$(
cat <<'SQL'
SELECT CAST(multiIf(number = 0, '{}', number = 1, '{"mix":[1,"user_1",null]}', number = 2, '{"mix":13}', '{"mix":[]}') AS JSON) AS j
FROM numbers(4)
FORMAT Native
SQL
)
capture "json_mixed_rows_rev0.native" "${json_mixed_rows_query}"
capture "json_shared_rev0.native" "${json_shared_query}"
capture "json_shared_rev54460.native" "${json_shared_query}" \
--data-urlencode "client_protocol_version=54460"
capture "json_shared_rev54485.native" "${json_shared_query}" \
--data-urlencode "client_protocol_version=54485"

json_table="ch_core_json_shared_fixture_$$"
execute() {
curl -sS --fail "${url}" --user "${user}:${password}" --data-binary "$1"
}
cleanup_json_table() {
execute "DROP TABLE IF EXISTS ${json_table}" >/dev/null
}
trap cleanup_json_table EXIT
execute "CREATE TABLE ${json_table} (id UInt8, d Dynamic) ENGINE=MergeTree ORDER BY id"
execute "SYSTEM STOP MERGES ${json_table}"
execute "INSERT INTO ${json_table} SELECT 0, '{\"n\":{\"x\":13}}'::JSON"
execute "INSERT INTO ${json_table} SELECT 1, ['{\"k\":1}'::JSON]"
json_multipart_query="SELECT id, d FROM ${json_table} ORDER BY id FORMAT Native"
capture "json_multipart_rev0.native" "${json_multipart_query}"
capture "json_multipart_rev54460.native" "${json_multipart_query}" \
--data-urlencode "client_protocol_version=54460"
capture "json_multipart_rev54485.native" "${json_multipart_query}" \
--data-urlencode "client_protocol_version=54485"
cleanup_json_table
trap - EXIT

echo "Wrote fixtures to ${fixture_dir}"
Loading
Loading