Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,8 @@ encode are at type parity across all of this: whole-buffer and streaming decode,
Arrow C Data export, and encode back to Native block bytes for the HTTP `INSERT`
path. The `compression` module reads LZ4/ZSTD/NONE frames and writes LZ4 frames;
the sans-I/O TCP engine in the sibling `ch-tcp-proto` crate covers the
handshake, Ping, SELECT with optional compression, and uncompressed INSERT at
revision 54460. Compressed INSERT and cancellation over TCP remain future work.
handshake, Ping, SELECT and INSERT with optional compression at revision 54460.
Cancellation over TCP remains future work.
The growth model is fixed: implement a type once here, following the documented
confirm-against-server-source workflow in `AGENTS.md`, and every binding gets it
for free. The roadmap lives in `README.md`.
14 changes: 12 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,17 @@ version; patch releases remain backward compatible.

## Unreleased

- `ch-tcp-proto`: support compressed client-fed INSERT. Upload blocks and the
empty upload terminator use LZ4 frames, with at most 1 MiB of plaintext per
frame. Server schema blocks use the existing compressed receive path.
`CompressedInsertUnsupported` is deprecated and no longer returned.
Captured revision-54460 exchanges cover engine-written multi-frame uploads,
server rejection, and recovery from errors before the upload schema.
- Changed: `ch-tcp-proto` Exceptions before `InsertSchema` now leave the
connection `Ready` instead of `Failed`, for compressed and uncompressed
INSERT. The request must have been taken and flushed, and the ordinary query
Exception rules still apply. Exceptions during or after upload remain
terminal, including after `finish_insert`.
- Fix `encode_block` rejecting `SimpleAggregateFunction(f, Nullable(T))` inside
Array, Tuple, Map, Nested, and Variant with a mismatched-column error. Alias
chains below Nullable now encode and decode in any position. Type headers and
Expand Down Expand Up @@ -41,8 +52,7 @@ version; patch releases remain backward compatible.
method byte. Plaintext size is checked against `max_block_bytes`, and frame
length against the worst-case compressed size for that budget, before the
payload is buffered. A bad frame fails the connection with
`ProtocolError::Compression`. `insert` on a compressed connection returns
`ProtocolError::CompressedInsertUnsupported` and queues nothing. Real server
`ProtocolError::Compression`. Real server
fixtures at revision 54460 cover LZ4, NONE and ZSTD, multi-frame blocks,
Totals and Extremes, a mid-stream exception, cancel, and reuse.
- Add `native::binary_value::write_binary_type_prefix` and
Expand Down
6 changes: 5 additions & 1 deletion CODEC_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,11 @@ The compression flag is sent on every Query. The empty client Data block
after Query is LZ4-framed too, as required by `TCPHandler::initBlockInput`
and `processData` in `src/Server/TCPHandler.cpp` at the same tag. The engine
does not send `network_compression_method` and accepts each supported method
per frame. Client-fed compressed INSERT remains unsupported.
per frame. Client-fed INSERT writes LZ4 frames with at most 1 MiB of plaintext
each, flushes at each Native block boundary, and compresses the upload
terminator too. Server schema Data uses the same compressed receive path.
This matches `Connection::sendData` in `src/Client/Connection.cpp` at
`v26.8.1.2041-lts`.

HTTP output normally auto-flushes after each chunk, so captured
Native frames align with blocks. Confirmed HTTP response buffering through
Expand Down
16 changes: 8 additions & 8 deletions COMPLETENESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,12 @@ default; the user may override it.
`encode_block` allocation as material. Further open-ended
`AggregateFunction` work also remains paused because each signature needs a
separately confirmed unframed state-boundary codec.
- **Recommended next:** wire compressed frames into the sibling packet engine.
Core now decodes every method `network_compression_method` accepts at
`v26.8.1.2041-lts`: NONE, LZ4, LZ4HC (the same 0x82 byte), and ZSTD. Its
revision-54460 uncompressed SELECT and INSERT support is complete, and real-server
LZ4/ZSTD fixtures are present. Revision-54492 String size streams and
sparse/replicated serialization remain on the broader codec roadmap.
- **TCP compression checkpoint, 2026-11-17:** the sibling packet engine supports
compressed SELECT and INSERT at revision 54460. Outbound blocks use LZ4.
Inbound frames accept NONE, LZ4/LZ4HC, and ZSTD. Captured INSERT success,
rejection, and recovery before the schema match `v26.8.1.2041-lts`. Cancellation,
revision-54492 String size streams, and sparse/replicated serialization remain
on the broader roadmap.
- **Key references:** the MultiPoint and Geometry contract is in
`CODEC_CONTRACT.md`. The logical and column models are in
`src/schema.rs` and `src/column.rs`; parser, binary descriptor, decode/scan,
Expand Down Expand Up @@ -1127,8 +1127,8 @@ is a separate sibling crate, not part of this codec's type-completeness bar.
Tracked elsewhere (README roadmap), intentionally not part of this tracker:

- The native TCP protocol engine, `crates/ch-tcp-proto`: handshake, Ping, and
uncompressed SELECT and INSERT at revision 54460; compression and
cancellation are tracked in its README and the root README roadmap.
SELECT and INSERT with optional compression at revision 54460. Cancellation
is tracked in its README and the root README roadmap.
- Per-runtime zero-copy adapters in the bindings (those live in the client
repos).
- Row-major materialization (a binding concern by design).
19 changes: 12 additions & 7 deletions INTEGRATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,11 +129,11 @@ dispatch. It reports Progress, ProfileInfo, Log, ProfileEvents, Totals,
Extremes, Exception, and EndOfStream as typed events. It does not own sockets,
TLS, timeouts, pooling, or cancellation: there is no Cancel packet yet, so a
driver that abandons a query mid-stream must discard the connection. The first
engine handles SELECT with optional compression and uncompressed INSERT.
Use `Connection::with_compression(true)` to enable compressed SELECT responses.
engine handles SELECT and INSERT with optional compression.
Use `Connection::with_compression(true)` to enable it for both directions.
The engine decodes NONE, LZ4, and ZSTD frames without choosing the server's
codec. Client-fed INSERT on that connection returns
`ProtocolError::CompressedInsertUnsupported`.
codec. Outbound INSERT blocks use LZ4 frames with at most 1 MiB of plaintext
each. Packet codes and table names stay raw.
See `crates/ch-tcp-proto/README.md`
for the driver loop and a tested `run_select` example.

Expand Down Expand Up @@ -167,7 +167,12 @@ partial or queued Log/ProfileEvents packets. Match column names and types to
the schema, since the server can reorder columns, fill omitted columns, drop
unknown columns, and coerce mismatched types. TableColumns metadata preserves
raw bytes, including non-UTF-8 literals. Its retention cap can truncate the
metadata without failing the upload. The engine stays sans-I/O throughout.
metadata without failing the upload. An Exception after the schema fails the
connection, including after `finish_insert`. Before the schema, the ordinary
query Exception reuse rules apply once the initial request has been taken
and flushed. All transferred output must reach the socket in order before
a later Query. The engine stays
sans-I/O throughout.

Each `Event::Data` payload is exactly one complete Native block at the
connection's `effective_revision()`, and the payloads concatenate into one
Expand Down Expand Up @@ -401,8 +406,8 @@ Send those bytes as the body of an `INSERT INTO t FORMAT Native` request. Notes
preamble and no per-column marker, and the stream ends at EOF. For TCP, use
the connection's effective revision. Frame each encoded block separately
when compression is enabled, including the empty end marker. Packet codes
and block names stay outside the compressed envelope. Transport and packet
framing remain the client's responsibility.
and block names stay outside the compressed envelope. `ch-tcp-proto` handles
packet framing through its INSERT API. Drivers own transport.
- Coverage: encode is at full type parity with decode. Every type the crate decodes it also encodes, so a batch the core produced round-trips back to Native bytes. Only a type the decoder itself does not support returns `EncodeError::UnsupportedType`.
- Validation: the input is your in-memory buffers, not wire bytes, but encode still validates the whole batch before writing anything and returns `EncodeError::InconsistentBatch` on a malformed column (wrong length, bad offsets, width mismatch, out-of-range `LowCardinality` index, a null in a non-nullable column). A rejected batch produces no partial bytes.
- Round trip: `decode(encode(x))` reproduces the buffers. The "Encoding" section of `CODEC_CONTRACT.md` has the precise preconditions, encoder choices, and round-trip guarantees.
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ Implemented:

Not implemented yet:

- Compressed TCP INSERT exchanges and query cancellation over TCP.
- Query cancellation over TCP.
- Language-specific materialization policy (bindings own this, by design).

## Supported types
Expand Down Expand Up @@ -164,9 +164,9 @@ In rough priority order:
`ArrayBuffer`, NumPy export that does not route through Arrow.
3. Insert-path completion: a streaming/sink encode API. Type parity is kept as
each decoder type lands.
4. Native TCP protocol engine, the rest of it: compressed INSERT exchanges
and cancellation. The handshake, Ping, SELECT with optional compression,
and uncompressed INSERT at revision 54460 ship in `ch-tcp-proto`.
4. Native TCP protocol engine, the rest of it: cancellation. The handshake,
Ping, SELECT and INSERT with optional compression at revision 54460 ship
in `ch-tcp-proto`.

## Consuming

Expand Down Expand Up @@ -216,8 +216,8 @@ block bytes for `INSERT` with `native::encode::encode_block` or

The root package remains `ch-core-rs`. The workspace also contains
[`ch-tcp-proto`](crates/ch-tcp-proto/README.md), a sans-I/O TCP protocol engine.
Its current scope is the revision-54460 handshake, Ping/Pong, SELECT with
optional compression, and uncompressed INSERT exchanges with Native byte output
Its current scope is the revision-54460 handshake, Ping/Pong, SELECT and INSERT
with optional compression, Native byte output
and query metadata events.
The engine depends on core.

Expand Down Expand Up @@ -258,7 +258,7 @@ version and first-byte hexdumps.

A working core under active development. The API is not yet stable. Decode and
encode are at type parity, both verified against a live ClickHouse server. Not
yet implemented: compressed TCP INSERT and cancellation; a
yet implemented: TCP cancellation; a
streaming/sink encode API is future work. Releases are tagged and semver'd, each
recorded in `CHANGELOG.md`. `ARCHITECTURE.md` describes how the pieces fit
together.
Expand Down
75 changes: 45 additions & 30 deletions crates/ch-tcp-proto/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,25 @@ sockets, TLS, timeouts, cancellation, and connection pooling. The engine
accepts arbitrary byte chunks, queues protocol events, and queues ordered bytes
to send.

The initial scope is deliberately small: handshake, Ping/Pong, SELECT with
optional compression, and uncompressed INSERT at protocol revision 54460.
The initial scope is deliberately small: handshake, Ping/Pong, SELECT and
INSERT with optional compression at protocol revision 54460.
SELECT results emit complete Native block bytes plus typed Progress, ProfileInfo, ProfileEvents, Log, Totals,
Extremes, Exception, and EndOfStream events. INSERT emits its schema separately
before accepting client blocks. Cancellation, external tables, and compressed
INSERT are not implemented yet.
before accepting client blocks. Cancellation and external tables are not
implemented yet.

Enable compression with `Connection::new(hello)?.with_compression(true)`.
The engine writes the compression flag into every Query and compresses the
empty external-data marker with LZ4. Server Data, Totals, and Extremes bodies
are decompressed before their Native bytes are emitted. Each frame selects its
own codec: NONE, LZ4, or ZSTD. The engine does not send
`network_compression_method`; server settings choose the response codec.
`insert` on a compressed connection returns
`ProtocolError::CompressedInsertUnsupported` before queuing any bytes.
Client-fed INSERT blocks use LZ4 frames with at most 1 MiB of plaintext each.
Both the external-data marker and the upload terminator are compressed. The
server schema block is decompressed before `InsertSchema` is emitted. Outbound
ZSTD is deferred because core only writes LZ4 frames. Response ZSTD remains
supported. `CompressedInsertUnsupported` is deprecated and retained for source
compatibility, but is no longer returned.

## Driver loop

Expand Down Expand Up @@ -377,24 +381,28 @@ Call `finish_insert()` once to queue the empty Data end marker and enter
`FinishingInsert`. Flush it, then keep feeding and draining through
`Event::EndOfStream`. Log and ProfileEvents may arrive during the exchange. An
Exception at any phase is an `Event::Exception`; stop uploading and report it.
Exceptions in an exchange started with `insert` fail the connection and discard
queued upload bytes. Protocol failures, including a failed `finish()`, also
discard queued output not yet taken, in every state including the handshake.
Discard the socket after either failure. After the external-table marker has
been consumed, an upload-phase Exception can leave the server draining client
data. Errors before the schema or after the upload terminator do not require
that upload drain. The engine conservatively retires the connection in all
`insert` error phases. There is no successful EndOfStream after an Exception.
INSERT Exceptions discard queued upload bytes. An Exception before the schema
permits reuse when all queued output has been taken and flushed, subject to
the terminal-code and first-response settings rules below. An Exception after
the schema always retires the connection, including after `finish_insert`.
The server may have stopped reading inside a block, so sending the terminator
does not establish safe alignment. An Exception while the initial request
remains queued also retires it. Protocol failures, including a
failed `finish()`, discard queued output and fail the connection. Discard the
socket whenever the state is `Failed`. There is no successful EndOfStream
after an Exception.
Drain all events even when `feed` reports a protocol error. `finish()` still
means transport EOF, not end of INSERT input.

Outbound blocks use one scan and one copy into the output queue. The engine
does not materialize rows or re-encode values. Take output between blocks to
bound queued upload memory. After consuming `InsertSchema`, upload and finish
calls can proceed while Log or ProfileEvents packets are partially received or
queued. Drivers should still feed and drain incoming events regularly. A
partly received Exception or unidentified packet blocks upload until it is
parsed, so a known pending failure cannot be bypassed by another upload call.
Uncompressed outbound blocks use one scan and one copy into the output queue.
Compressed blocks use the same scan, then write LZ4 frames into the queue. The
engine does not materialize rows or re-encode values. Take output between
blocks to bound queued upload memory. After consuming `InsertSchema`, upload
and finish calls can proceed while Log or ProfileEvents packets are partially
received or queued. Drivers should still feed and drain incoming events
regularly. A partly received Exception or unidentified packet blocks upload
until it is parsed, so a known pending failure cannot be bypassed by another
upload call.

## Contract

Expand Down Expand Up @@ -434,9 +442,11 @@ parsed, so a known pending failure cannot be bypassed by another upload call.
- Once a query context exists, other query errors normally leave the
connection reusable. The wire does not identify every server throw site,
so a driver must still honor transport EOF and I/O failures.
- Exchanges started with `insert()` always fail the connection on Exception,
because upload drain completion cannot be established from the Exception
packet alone. Queued upload bytes are discarded.
- INSERT Exceptions during `Inserting` or `FinishingInsert`, or while output
remains queued, fail the connection. Before the schema, the same reuse
rules as SELECT apply. Queued upload bytes are discarded.
`take_output` transfers ownership to the driver, which must flush all
transferred bytes in order before reusing the connection.
- There is no Cancel packet and no reset. A driver that abandons a query before
its EndOfStream or Exception, for example on a timeout, must discard the
`Connection` and its socket rather than reuse them.
Expand All @@ -461,8 +471,7 @@ parsed, so a known pending failure cannot be bypassed by another upload call.
- Query support is limited to statements whose complete response uses the
initial SELECT result exchange, including fully specified inline INSERTs
and INSERT SELECT. Use `insert` for a client-fed INSERT.
input(), external tables,
parallel-replica exchanges, cancellation, and compressed INSERT are not
input(), external tables, parallel-replica exchanges and cancellation are not
supported. The engine does not parse SQL to classify statements.
- The engine frames result blocks with core's scanner, so a result column type
core cannot decode fails the whole connection with `ProtocolError::Decode`.
Expand Down Expand Up @@ -563,8 +572,9 @@ Log, ProfileEvents, and TableColumns remain raw at revision 54460.
Core's scan-only boundary parser finds the block's exact end without allocating decoded
columns. Malformed inbound bodies surface as `ProtocolError::Decode` and fail
the connection; malformed outbound INSERT bodies return `InvalidInsertBlock`
and leave the connection usable. Failure to reserve the outbound copy returns
`InsertOutputAllocationFailed` before changing state or output.
and leave the connection usable. Both uncompressed and compressed uploads
reserve the entire packet before writing. Allocation failures return
`InsertOutputAllocationFailed` and leave state and output unchanged.

Framing retains progress through column bodies and runtime type-table
prefixes, including Dynamic, JSON, AggregateFunction, and nested containers.
Expand All @@ -585,8 +595,13 @@ The INSERT sequence is confirmed at `v26.8.1.2041-lts` in
`src/Server/TCPHandler.cpp`, `TCPHandler::{readTemporaryTables,startInsertQuery,processInsertQuery,receivePacketsExpectData,processData,sendTableColumns,sendEndOfStream}`.
`NativeWriter::write` and `NativeReader::read` in `src/Formats/` define the
block body. The handler sends an upload Exception before `skipData` drains
remaining input during upload. The engine conservatively retires connections
started with `insert` on errors. Inline completion without a schema is confirmed in
remaining input during upload. `NativeReader::read` can throw before consuming
the whole block. `TCPHandler::processUnexpectedData` starts a fresh reader
while draining, so unread block bytes can prevent the drain from reaching a
sent terminator. After-schema Exceptions therefore retire the connection.
The before-schema recovery rule is inferred from the initial external-table
marker handling and verified by compressed and uncompressed live captures.
Inline completion without a schema is confirmed in
`src/Interpreters/InterpreterInsertQuery.cpp`,
`InterpreterInsertQuery::{buildInsertPipeline,buildInsertSelectPipeline}`,
and the completed-pipeline dispatch in `TCPHandler.cpp`.
Expand Down
Loading
Loading