diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b91e27c..cb805b6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -424,8 +424,8 @@ encode are at type parity across all of this: whole-buffer and streaming decode, Arrow C Data export, and encode back to Native block bytes for the HTTP `INSERT` path. The `compression` module reads LZ4/ZSTD/NONE frames and writes LZ4 frames; the sans-I/O TCP engine in the sibling `ch-tcp-proto` crate covers the -handshake, Ping, SELECT with optional compression, and uncompressed INSERT at -revision 54460. Compressed INSERT and cancellation over TCP remain future work. +handshake, Ping, SELECT and INSERT with optional compression at revision 54460. +Cancellation over TCP remains future work. The growth model is fixed: implement a type once here, following the documented confirm-against-server-source workflow in `AGENTS.md`, and every binding gets it for free. The roadmap lives in `README.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index c696a24..6dcf066 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ version; patch releases remain backward compatible. ## Unreleased +- `ch-tcp-proto`: support compressed client-fed INSERT. Upload blocks and the + empty upload terminator use LZ4 frames, with at most 1 MiB of plaintext per + frame. Server schema blocks use the existing compressed receive path. + `CompressedInsertUnsupported` is deprecated and no longer returned. + Captured revision-54460 exchanges cover engine-written multi-frame uploads, + server rejection, and recovery from errors before the upload schema. +- Changed: `ch-tcp-proto` Exceptions before `InsertSchema` now leave the + connection `Ready` instead of `Failed`, for compressed and uncompressed + INSERT. The request must have been taken and flushed, and the ordinary query + Exception rules still apply. Exceptions during or after upload remain + terminal, including after `finish_insert`. - Fix `encode_block` rejecting `SimpleAggregateFunction(f, Nullable(T))` inside Array, Tuple, Map, Nested, and Variant with a mismatched-column error. Alias chains below Nullable now encode and decode in any position. Type headers and @@ -41,8 +52,7 @@ version; patch releases remain backward compatible. method byte. Plaintext size is checked against `max_block_bytes`, and frame length against the worst-case compressed size for that budget, before the payload is buffered. A bad frame fails the connection with - `ProtocolError::Compression`. `insert` on a compressed connection returns - `ProtocolError::CompressedInsertUnsupported` and queues nothing. Real server + `ProtocolError::Compression`. Real server fixtures at revision 54460 cover LZ4, NONE and ZSTD, multi-frame blocks, Totals and Extremes, a mid-stream exception, cancel, and reuse. - Add `native::binary_value::write_binary_type_prefix` and diff --git a/CODEC_CONTRACT.md b/CODEC_CONTRACT.md index 610eae2..9970fdc 100644 --- a/CODEC_CONTRACT.md +++ b/CODEC_CONTRACT.md @@ -163,7 +163,11 @@ The compression flag is sent on every Query. The empty client Data block after Query is LZ4-framed too, as required by `TCPHandler::initBlockInput` and `processData` in `src/Server/TCPHandler.cpp` at the same tag. The engine does not send `network_compression_method` and accepts each supported method -per frame. Client-fed compressed INSERT remains unsupported. +per frame. Client-fed INSERT writes LZ4 frames with at most 1 MiB of plaintext +each, flushes at each Native block boundary, and compresses the upload +terminator too. Server schema Data uses the same compressed receive path. +This matches `Connection::sendData` in `src/Client/Connection.cpp` at +`v26.8.1.2041-lts`. HTTP output normally auto-flushes after each chunk, so captured Native frames align with blocks. Confirmed HTTP response buffering through diff --git a/COMPLETENESS.md b/COMPLETENESS.md index 85d994f..d3b39c5 100644 --- a/COMPLETENESS.md +++ b/COMPLETENESS.md @@ -92,12 +92,12 @@ default; the user may override it. `encode_block` allocation as material. Further open-ended `AggregateFunction` work also remains paused because each signature needs a separately confirmed unframed state-boundary codec. -- **Recommended next:** wire compressed frames into the sibling packet engine. - Core now decodes every method `network_compression_method` accepts at - `v26.8.1.2041-lts`: NONE, LZ4, LZ4HC (the same 0x82 byte), and ZSTD. Its - revision-54460 uncompressed SELECT and INSERT support is complete, and real-server - LZ4/ZSTD fixtures are present. Revision-54492 String size streams and - sparse/replicated serialization remain on the broader codec roadmap. +- **TCP compression checkpoint, 2026-11-17:** the sibling packet engine supports + compressed SELECT and INSERT at revision 54460. Outbound blocks use LZ4. + Inbound frames accept NONE, LZ4/LZ4HC, and ZSTD. Captured INSERT success, + rejection, and recovery before the schema match `v26.8.1.2041-lts`. Cancellation, + revision-54492 String size streams, and sparse/replicated serialization remain + on the broader roadmap. - **Key references:** the MultiPoint and Geometry contract is in `CODEC_CONTRACT.md`. The logical and column models are in `src/schema.rs` and `src/column.rs`; parser, binary descriptor, decode/scan, @@ -1127,8 +1127,8 @@ is a separate sibling crate, not part of this codec's type-completeness bar. Tracked elsewhere (README roadmap), intentionally not part of this tracker: - The native TCP protocol engine, `crates/ch-tcp-proto`: handshake, Ping, and - uncompressed SELECT and INSERT at revision 54460; compression and - cancellation are tracked in its README and the root README roadmap. + SELECT and INSERT with optional compression at revision 54460. Cancellation + is tracked in its README and the root README roadmap. - Per-runtime zero-copy adapters in the bindings (those live in the client repos). - Row-major materialization (a binding concern by design). diff --git a/INTEGRATING.md b/INTEGRATING.md index 4bae58b..1c4d0f5 100644 --- a/INTEGRATING.md +++ b/INTEGRATING.md @@ -129,11 +129,11 @@ dispatch. It reports Progress, ProfileInfo, Log, ProfileEvents, Totals, Extremes, Exception, and EndOfStream as typed events. It does not own sockets, TLS, timeouts, pooling, or cancellation: there is no Cancel packet yet, so a driver that abandons a query mid-stream must discard the connection. The first -engine handles SELECT with optional compression and uncompressed INSERT. -Use `Connection::with_compression(true)` to enable compressed SELECT responses. +engine handles SELECT and INSERT with optional compression. +Use `Connection::with_compression(true)` to enable it for both directions. The engine decodes NONE, LZ4, and ZSTD frames without choosing the server's -codec. Client-fed INSERT on that connection returns -`ProtocolError::CompressedInsertUnsupported`. +codec. Outbound INSERT blocks use LZ4 frames with at most 1 MiB of plaintext +each. Packet codes and table names stay raw. See `crates/ch-tcp-proto/README.md` for the driver loop and a tested `run_select` example. @@ -167,7 +167,12 @@ partial or queued Log/ProfileEvents packets. Match column names and types to the schema, since the server can reorder columns, fill omitted columns, drop unknown columns, and coerce mismatched types. TableColumns metadata preserves raw bytes, including non-UTF-8 literals. Its retention cap can truncate the -metadata without failing the upload. The engine stays sans-I/O throughout. +metadata without failing the upload. An Exception after the schema fails the +connection, including after `finish_insert`. Before the schema, the ordinary +query Exception reuse rules apply once the initial request has been taken +and flushed. All transferred output must reach the socket in order before +a later Query. The engine stays +sans-I/O throughout. Each `Event::Data` payload is exactly one complete Native block at the connection's `effective_revision()`, and the payloads concatenate into one @@ -401,8 +406,8 @@ Send those bytes as the body of an `INSERT INTO t FORMAT Native` request. Notes preamble and no per-column marker, and the stream ends at EOF. For TCP, use the connection's effective revision. Frame each encoded block separately when compression is enabled, including the empty end marker. Packet codes - and block names stay outside the compressed envelope. Transport and packet - framing remain the client's responsibility. + and block names stay outside the compressed envelope. `ch-tcp-proto` handles + packet framing through its INSERT API. Drivers own transport. - Coverage: encode is at full type parity with decode. Every type the crate decodes it also encodes, so a batch the core produced round-trips back to Native bytes. Only a type the decoder itself does not support returns `EncodeError::UnsupportedType`. - Validation: the input is your in-memory buffers, not wire bytes, but encode still validates the whole batch before writing anything and returns `EncodeError::InconsistentBatch` on a malformed column (wrong length, bad offsets, width mismatch, out-of-range `LowCardinality` index, a null in a non-nullable column). A rejected batch produces no partial bytes. - Round trip: `decode(encode(x))` reproduces the buffers. The "Encoding" section of `CODEC_CONTRACT.md` has the precise preconditions, encoder choices, and round-trip guarantees. diff --git a/README.md b/README.md index 06b1efd..6ffbe58 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,7 @@ Implemented: Not implemented yet: -- Compressed TCP INSERT exchanges and query cancellation over TCP. +- Query cancellation over TCP. - Language-specific materialization policy (bindings own this, by design). ## Supported types @@ -164,9 +164,9 @@ In rough priority order: `ArrayBuffer`, NumPy export that does not route through Arrow. 3. Insert-path completion: a streaming/sink encode API. Type parity is kept as each decoder type lands. -4. Native TCP protocol engine, the rest of it: compressed INSERT exchanges - and cancellation. The handshake, Ping, SELECT with optional compression, - and uncompressed INSERT at revision 54460 ship in `ch-tcp-proto`. +4. Native TCP protocol engine, the rest of it: cancellation. The handshake, + Ping, SELECT and INSERT with optional compression at revision 54460 ship + in `ch-tcp-proto`. ## Consuming @@ -216,8 +216,8 @@ block bytes for `INSERT` with `native::encode::encode_block` or The root package remains `ch-core-rs`. The workspace also contains [`ch-tcp-proto`](crates/ch-tcp-proto/README.md), a sans-I/O TCP protocol engine. -Its current scope is the revision-54460 handshake, Ping/Pong, SELECT with -optional compression, and uncompressed INSERT exchanges with Native byte output +Its current scope is the revision-54460 handshake, Ping/Pong, SELECT and INSERT +with optional compression, Native byte output and query metadata events. The engine depends on core. @@ -258,7 +258,7 @@ version and first-byte hexdumps. A working core under active development. The API is not yet stable. Decode and encode are at type parity, both verified against a live ClickHouse server. Not -yet implemented: compressed TCP INSERT and cancellation; a +yet implemented: TCP cancellation; a streaming/sink encode API is future work. Releases are tagged and semver'd, each recorded in `CHANGELOG.md`. `ARCHITECTURE.md` describes how the pieces fit together. diff --git a/crates/ch-tcp-proto/README.md b/crates/ch-tcp-proto/README.md index fcd482b..9443b1a 100644 --- a/crates/ch-tcp-proto/README.md +++ b/crates/ch-tcp-proto/README.md @@ -6,12 +6,12 @@ sockets, TLS, timeouts, cancellation, and connection pooling. The engine accepts arbitrary byte chunks, queues protocol events, and queues ordered bytes to send. -The initial scope is deliberately small: handshake, Ping/Pong, SELECT with -optional compression, and uncompressed INSERT at protocol revision 54460. +The initial scope is deliberately small: handshake, Ping/Pong, SELECT and +INSERT with optional compression at protocol revision 54460. SELECT results emit complete Native block bytes plus typed Progress, ProfileInfo, ProfileEvents, Log, Totals, Extremes, Exception, and EndOfStream events. INSERT emits its schema separately -before accepting client blocks. Cancellation, external tables, and compressed -INSERT are not implemented yet. +before accepting client blocks. Cancellation and external tables are not +implemented yet. Enable compression with `Connection::new(hello)?.with_compression(true)`. The engine writes the compression flag into every Query and compresses the @@ -19,8 +19,12 @@ empty external-data marker with LZ4. Server Data, Totals, and Extremes bodies are decompressed before their Native bytes are emitted. Each frame selects its own codec: NONE, LZ4, or ZSTD. The engine does not send `network_compression_method`; server settings choose the response codec. -`insert` on a compressed connection returns -`ProtocolError::CompressedInsertUnsupported` before queuing any bytes. +Client-fed INSERT blocks use LZ4 frames with at most 1 MiB of plaintext each. +Both the external-data marker and the upload terminator are compressed. The +server schema block is decompressed before `InsertSchema` is emitted. Outbound +ZSTD is deferred because core only writes LZ4 frames. Response ZSTD remains +supported. `CompressedInsertUnsupported` is deprecated and retained for source +compatibility, but is no longer returned. ## Driver loop @@ -377,24 +381,28 @@ Call `finish_insert()` once to queue the empty Data end marker and enter `FinishingInsert`. Flush it, then keep feeding and draining through `Event::EndOfStream`. Log and ProfileEvents may arrive during the exchange. An Exception at any phase is an `Event::Exception`; stop uploading and report it. -Exceptions in an exchange started with `insert` fail the connection and discard -queued upload bytes. Protocol failures, including a failed `finish()`, also -discard queued output not yet taken, in every state including the handshake. -Discard the socket after either failure. After the external-table marker has -been consumed, an upload-phase Exception can leave the server draining client -data. Errors before the schema or after the upload terminator do not require -that upload drain. The engine conservatively retires the connection in all -`insert` error phases. There is no successful EndOfStream after an Exception. +INSERT Exceptions discard queued upload bytes. An Exception before the schema +permits reuse when all queued output has been taken and flushed, subject to +the terminal-code and first-response settings rules below. An Exception after +the schema always retires the connection, including after `finish_insert`. +The server may have stopped reading inside a block, so sending the terminator +does not establish safe alignment. An Exception while the initial request +remains queued also retires it. Protocol failures, including a +failed `finish()`, discard queued output and fail the connection. Discard the +socket whenever the state is `Failed`. There is no successful EndOfStream +after an Exception. Drain all events even when `feed` reports a protocol error. `finish()` still means transport EOF, not end of INSERT input. -Outbound blocks use one scan and one copy into the output queue. The engine -does not materialize rows or re-encode values. Take output between blocks to -bound queued upload memory. After consuming `InsertSchema`, upload and finish -calls can proceed while Log or ProfileEvents packets are partially received or -queued. Drivers should still feed and drain incoming events regularly. A -partly received Exception or unidentified packet blocks upload until it is -parsed, so a known pending failure cannot be bypassed by another upload call. +Uncompressed outbound blocks use one scan and one copy into the output queue. +Compressed blocks use the same scan, then write LZ4 frames into the queue. The +engine does not materialize rows or re-encode values. Take output between +blocks to bound queued upload memory. After consuming `InsertSchema`, upload +and finish calls can proceed while Log or ProfileEvents packets are partially +received or queued. Drivers should still feed and drain incoming events +regularly. A partly received Exception or unidentified packet blocks upload +until it is parsed, so a known pending failure cannot be bypassed by another +upload call. ## Contract @@ -434,9 +442,11 @@ parsed, so a known pending failure cannot be bypassed by another upload call. - Once a query context exists, other query errors normally leave the connection reusable. The wire does not identify every server throw site, so a driver must still honor transport EOF and I/O failures. - - Exchanges started with `insert()` always fail the connection on Exception, - because upload drain completion cannot be established from the Exception - packet alone. Queued upload bytes are discarded. + - INSERT Exceptions during `Inserting` or `FinishingInsert`, or while output + remains queued, fail the connection. Before the schema, the same reuse + rules as SELECT apply. Queued upload bytes are discarded. + `take_output` transfers ownership to the driver, which must flush all + transferred bytes in order before reusing the connection. - There is no Cancel packet and no reset. A driver that abandons a query before its EndOfStream or Exception, for example on a timeout, must discard the `Connection` and its socket rather than reuse them. @@ -461,8 +471,7 @@ parsed, so a known pending failure cannot be bypassed by another upload call. - Query support is limited to statements whose complete response uses the initial SELECT result exchange, including fully specified inline INSERTs and INSERT SELECT. Use `insert` for a client-fed INSERT. - input(), external tables, - parallel-replica exchanges, cancellation, and compressed INSERT are not + input(), external tables, parallel-replica exchanges and cancellation are not supported. The engine does not parse SQL to classify statements. - The engine frames result blocks with core's scanner, so a result column type core cannot decode fails the whole connection with `ProtocolError::Decode`. @@ -563,8 +572,9 @@ Log, ProfileEvents, and TableColumns remain raw at revision 54460. Core's scan-only boundary parser finds the block's exact end without allocating decoded columns. Malformed inbound bodies surface as `ProtocolError::Decode` and fail the connection; malformed outbound INSERT bodies return `InvalidInsertBlock` -and leave the connection usable. Failure to reserve the outbound copy returns -`InsertOutputAllocationFailed` before changing state or output. +and leave the connection usable. Both uncompressed and compressed uploads +reserve the entire packet before writing. Allocation failures return +`InsertOutputAllocationFailed` and leave state and output unchanged. Framing retains progress through column bodies and runtime type-table prefixes, including Dynamic, JSON, AggregateFunction, and nested containers. @@ -585,8 +595,13 @@ The INSERT sequence is confirmed at `v26.8.1.2041-lts` in `src/Server/TCPHandler.cpp`, `TCPHandler::{readTemporaryTables,startInsertQuery,processInsertQuery,receivePacketsExpectData,processData,sendTableColumns,sendEndOfStream}`. `NativeWriter::write` and `NativeReader::read` in `src/Formats/` define the block body. The handler sends an upload Exception before `skipData` drains -remaining input during upload. The engine conservatively retires connections -started with `insert` on errors. Inline completion without a schema is confirmed in +remaining input during upload. `NativeReader::read` can throw before consuming +the whole block. `TCPHandler::processUnexpectedData` starts a fresh reader +while draining, so unread block bytes can prevent the drain from reaching a +sent terminator. After-schema Exceptions therefore retire the connection. +The before-schema recovery rule is inferred from the initial external-table +marker handling and verified by compressed and uncompressed live captures. +Inline completion without a schema is confirmed in `src/Interpreters/InterpreterInsertQuery.cpp`, `InterpreterInsertQuery::{buildInsertPipeline,buildInsertSelectPipeline}`, and the completed-pipeline dispatch in `TCPHandler.cpp`. diff --git a/crates/ch-tcp-proto/examples/gen_compressed_insert_fixture_client.rs b/crates/ch-tcp-proto/examples/gen_compressed_insert_fixture_client.rs new file mode 100644 index 0000000..f08f8ba --- /dev/null +++ b/crates/ch-tcp-proto/examples/gen_compressed_insert_fixture_client.rs @@ -0,0 +1,145 @@ +//! Fixture tooling only for scripts/gen_tcp_compressed_insert_fixtures.py. + +use std::error::Error; +use std::fs; +use std::path::Path; + +use ch_core_rs::batch::ColBatch; +use ch_core_rs::column::{Column, PrimitiveColumn, Utf8Column}; +use ch_core_rs::compression::write_lz4_frame; +use ch_core_rs::native::encode::{encode_block, EncodeOptions}; +use ch_core_rs::schema::{ChType, Field, Schema}; +use ch_tcp_proto::{ClientHello, Connection, Event, Query, ENGINE_REVISION}; + +const HELLO: &[u8] = + include_bytes!("../../../tests/fixtures/tcp_handshake_server_hello_rev54460.packet"); +const TABLE: &str = "ch_core_tcp_compressed_insert_fixture"; + +fn query(sql: String, label: &str, settings: bool) -> Query { + let query = Query::new(sql) + .with_query_id(format!("ch-core-compressed-insert-{label}")) + .with_initial_address("192.0.2.13:79") + .with_initial_query_start_time_microseconds(1_791_500_000_000_013) + .with_os_user("fixture_os_user_13") + .with_hostname("fixture_host_79"); + if settings { + query + .with_setting("send_profile_events", "0") + .with_setting("send_logs_level", "none") + } else { + query + } +} + +fn batch(number: u64, value: &str, empty: bool) -> ColBatch { + let numbers = if empty { Vec::new() } else { vec![number] }; + let strings = if empty { + Utf8Column::new(vec![0], Vec::new()) + } else { + Utf8Column::new(vec![0, value.len() as i32], value.as_bytes().to_vec()) + }; + ColBatch::new( + Schema::new(vec![ + Field { + name: "n".into(), + ch_type: ChType::UInt64, + }, + Field { + name: "s".into(), + ch_type: ChType::String, + }, + ]), + vec![ + Column::UInt64(PrimitiveColumn::new(numbers)), + Column::Utf8(strings), + ], + usize::from(!empty), + ) +} + +fn ready(output: &Path, compressed: bool) -> Result> { + let hello = ClientHello::new("ch-core-query-fixture", "default") + .with_client_version(13, 79) + .with_quota_key("fixture_quota_13"); + let mut connection = Connection::new(hello)?.with_compression(compressed); + fs::write( + output.join("tcp_compressed_insert_client_hello.bytes"), + connection.take_output(), + )?; + connection.feed(HELLO)?; + assert!(matches!( + connection.poll_event(), + Some(Event::ServerHello(_)) + )); + fs::write( + output.join("tcp_compressed_insert_addendum.bytes"), + connection.take_output(), + )?; + Ok(connection) +} + +fn save( + output: &Path, + label: &str, + stage: &str, + connection: &mut Connection, +) -> Result<(), Box> { + let name = format!("tcp_compressed_insert_{label}_{stage}_rev54460.bytes"); + fs::write(output.join(name), connection.take_output())?; + Ok(()) +} + +fn main() -> Result<(), Box> { + let output = std::env::args_os() + .nth(1) + .ok_or("missing output directory")?; + let output = Path::new(&output); + fs::create_dir_all(output)?; + let mut options = EncodeOptions::default(); + options.protocol_revision = ENGINE_REVISION; + for label in ["multi", "rejected", "before_schema", "plain_before_schema"] { + let compressed = label != "plain_before_schema"; + let before_schema = matches!(label, "before_schema" | "plain_before_schema"); + let sql = if before_schema { + "INSERT INTO ch_core_tcp_compressed_insert_missing (n, s) VALUES".to_owned() + } else { + format!("INSERT INTO {TABLE} (n, s) VALUES") + }; + let mut connection = ready(output, compressed)?; + connection.insert(query(sql, label, !before_schema))?; + save(output, label, "request", &mut connection)?; + if !before_schema { + let native = encode_block(&batch(0, "", true), &options)?; + let mut schema = vec![1, 0]; + write_lz4_frame(&native, &mut schema)?; + connection.feed(&schema)?; + assert!(matches!( + connection.poll_event(), + Some(Event::InsertSchema(_)) + )); + let large = "x".repeat(1_100_000); + let rows = if label == "multi" { + vec![(13, "user_1"), (79, large.as_str())] + } else { + vec![(179, "rejected")] + }; + for (index, (number, value)) in rows.into_iter().enumerate() { + connection.insert_data(&encode_block(&batch(number, value, false), &options)?)?; + save(output, label, &format!("block{index}"), &mut connection)?; + } + connection.finish_insert()?; + save(output, label, "finish", &mut connection)?; + } + if label != "rejected" { + let mut recovery = ready(output, compressed)?; + let sql = if before_schema { + "SELECT toUInt64(13) AS n, 'user_1' AS s".to_owned() + } else { + format!("SELECT n, s FROM {TABLE} ORDER BY n") + }; + recovery.query(query(sql, &format!("{label}_recovery"), true))?; + save(output, label, "recovery_request", &mut recovery)?; + } + } + Ok(()) +} diff --git a/crates/ch-tcp-proto/src/lib.rs b/crates/ch-tcp-proto/src/lib.rs index 7059a3d..2d4288b 100644 --- a/crates/ch-tcp-proto/src/lib.rs +++ b/crates/ch-tcp-proto/src/lib.rs @@ -6,7 +6,8 @@ use std::fmt; use std::sync::Arc; use ch_core_rs::compression::{ - decompress_frame_into, frame_header, ChecksumMode, CompressionError, FrameHeader, + decompress_frame_into, frame_header, write_lz4_frame, ChecksumMode, CompressionError, + FrameHeader, }; use ch_core_rs::native::decode::{DecodeError, DecodeOptions}; use ch_core_rs::native::stream_decoder::{BlockBoundary, BlockScanner}; @@ -102,6 +103,8 @@ const EMPTY_NATIVE_BLOCK_LZ4: &[u8] = &[ ]; /// Checksum plus header bytes before a compression frame's payload. const FRAME_PREFIX_LEN: usize = 25; +/// Default `CompressedWriteBuffer` size at v26.8.1.2041-lts. +const INSERT_FRAME_PLAINTEXT: usize = 1024 * 1024; // Thresholds from `src/Core/ProtocolDefines.h` at v26.8.1.2041-lts. const MIN_REVISION_WITH_SERVER_TIMEZONE: u64 = 54_058; @@ -478,9 +481,11 @@ pub enum Event { /// for the connection-level query cases below, where the connection /// becomes [`State::Failed`] with [`FailureKind::ServerException`]. Other /// query errors return to [`State::Ready`] after this event is queued. - /// Every exchange started with [`Connection::insert`] conservatively - /// fails on Exception and discards queued upload bytes, because the server - /// may be waiting for the rest of an upload in its skip-data path. + /// Ordinary INSERT errors return to Ready before the schema, provided all + /// queued request bytes were taken by the driver. After the schema they + /// fail the connection because the server's Native reader may have stopped + /// inside an upload block. + /// Every INSERT Exception discards queued output. /// Either way it is the last event of its exchange: no EndOfStream, Log, /// or ProfileEvents follows it. /// @@ -510,6 +515,14 @@ pub enum Event { /// - Once the query context exists, other query errors normally leave the /// connection reusable. The wire does not identify every server throw /// site, so drivers must still honor transport EOF and I/O failures. + /// - `TCPHandler::runImpl` sends the Exception before `skipData` drains + /// any remaining client Data through the no-column marker. Before the + /// schema, the initial external-table marker suffices. After the schema, + /// the upload marker is required. `NativeReader::read` in + /// `src/Formats/NativeReader.cpp` can fail inside a block, while + /// `TCPHandler::processUnexpectedData` starts a fresh reader. The upload + /// marker does not establish safe alignment, so the engine retires the + /// connection after the schema. /// /// Boxed to keep [`Event`] small. Exception(Box), @@ -528,11 +541,11 @@ const _: () = assert!( #[non_exhaustive] pub enum FailureKind { /// The server sent a complete Exception packet in the handshake or - /// pre-query phase, a connection-level query Exception, or any INSERT - /// Exception. The server closes in the first cases. INSERT is treated as - /// terminal because safe reuse would require finishing or draining its - /// skip-data exchange. Bytes after the Exception are not parsed. See - /// [`Event::Exception`] for details. + /// pre-query phase, a connection-level query Exception, or an INSERT + /// Exception after the schema or before its request was transferred. The + /// server closes in the first cases. An INSERT reader may be misaligned + /// or still require draining its upload. Bytes after the Exception are not + /// parsed. See [`Event::Exception`] for details. ServerException, /// Malformed, unexpected, oversized, or truncated input. Protocol, @@ -651,16 +664,17 @@ pub enum ProtocolError { /// then fails with [`ProtocolError::Decode`] on the first block. Keep the /// setting disabled server-side. BinaryNativeTypeEncodingUnsupported, - /// Client-fed INSERT compression is not implemented yet. - /// - /// A connection configured with [`Connection::with_compression`] may run - /// queries, including inline INSERT forms that need no later client data, - /// but [`Connection::insert`] is rejected before writing anything. + /// Retained for compatibility. Compressed INSERT is supported and this + /// variant is no longer returned. + #[deprecated(note = "compressed INSERT is supported")] CompressedInsertUnsupported, - /// Core rejected a ClickHouse compression frame. + /// Core rejected or could not write a ClickHouse compression frame. /// - /// The connection fails because the next packet boundary cannot be found - /// safely. The [`CompressionError`] is also returned by [`Error::source`]. + /// An inbound error fails the connection because its next packet boundary + /// cannot be found safely. An outbound framing error leaves output and + /// state unchanged. Allocation failures from [`Connection::insert_data`] + /// use [`ProtocolError::InsertOutputAllocationFailed`]. The + /// [`CompressionError`] is also returned by [`Error::source`]. Compression(CompressionError), /// An inbound Native block exceeded the per-packet limit. Always fatal: /// the connection fails with [`FailureKind::Protocol`]. @@ -751,10 +765,11 @@ impl fmt::Display for ProtocolError { Self::BinaryNativeTypeEncodingUnsupported => formatter.write_str( "output_format_native_encode_types_in_binary_format accepts only \"0\"", ), + #[allow(deprecated)] Self::CompressedInsertUnsupported => { formatter.write_str("compressed client-fed INSERT is not supported") } - Self::Compression(_) => formatter.write_str("compression frame decode failed"), + Self::Compression(_) => formatter.write_str("compression frame failed"), Self::NativeBlockTooLarge { limit } => { write!(formatter, "Native block exceeds the {limit} byte limit") } @@ -855,6 +870,7 @@ impl PartialEq for ProtocolError { (InvalidInsertBlock(left), InvalidInsertBlock(right)) => left == right, (InvalidInsertSchema, InvalidInsertSchema) => true, (BinaryNativeTypeEncodingUnsupported, BinaryNativeTypeEncodingUnsupported) => true, + #[allow(deprecated)] (CompressedInsertUnsupported, CompressedInsertUnsupported) => true, (Compression(left), Compression(right)) => left == right, (NativeBlockTooLarge { limit: left }, NativeBlockTooLarge { limit: right }) => { @@ -1008,7 +1024,7 @@ struct ClientIdentity { } /// Incremental handshake, Ping/Pong, and Query engine with optional compressed -/// SELECT. +/// SELECT and INSERT. /// /// Events have exactly one channel: [`Self::feed`] reports only errors, and /// every completed packet is queued for [`Self::poll_event`]. A `Vec` return @@ -1218,10 +1234,9 @@ impl Connection { /// LZ4/LZ4HC, and ZSTD frame methods are accepted according to each frame's /// method byte. Packet codes and table names remain raw. /// - /// Client-fed compressed INSERT is deferred: [`Self::insert`] returns - /// [`ProtocolError::CompressedInsertUnsupported`] without changing the - /// connection. Inline INSERT forms sent through [`Self::query`] remain - /// supported because they need no later client Data packet. + /// INSERT schema blocks are decompressed through the same path. Client + /// upload blocks and their end marker use LZ4. Upload frames contain at + /// most 1 MiB of plaintext and never cross Native block boundaries. pub fn with_compression(mut self, enabled: bool) -> Self { self.compression = enabled; self @@ -1313,7 +1328,7 @@ impl Connection { ) } - /// Starts an uncompressed INSERT that may request client Native blocks. + /// Starts an INSERT that may request client Native blocks. /// /// The queued bytes are the Query packet followed by the empty Data block /// that terminates external tables. Drain the resulting @@ -1326,11 +1341,10 @@ impl Connection { /// The same Query validation and 64 MiB request limit as [`Self::query`] /// apply. A server Exception may end any INSERT phase. If one arrives, /// queued upload bytes are discarded so they cannot leak into a later - /// exchange. + /// exchange. Reuse is allowed before the schema only when all queued + /// output was taken, subject to the connection-level error rules in + /// [`Event::Exception`]. pub fn insert(&mut self, query: Query) -> Result<(), ProtocolError> { - if self.compression { - return Err(ProtocolError::CompressedInsertUnsupported); - } self.start_query( query, State::AwaitingInsertSchema, @@ -1409,16 +1423,11 @@ impl Connection { }); } } - let empty_data_body = if self.compression { - EMPTY_NATIVE_BLOCK_LZ4 - } else { - EMPTY_NATIVE_BLOCK - }; let packet_size = query_packet_len( &query, &self.identity, &self.quota_key, - empty_data_body.len(), + empty_native_block(self.compression).len(), ) .ok_or(ProtocolError::QueryOutputTooLarge { length: usize::MAX, @@ -1477,9 +1486,7 @@ impl Connection { } } put_string(packet, ""); - put_varuint(packet, CLIENT_PACKET_DATA); - put_string(packet, ""); - packet.extend_from_slice(empty_data_body); + put_empty_data_packet(packet, self.compression); debug_assert_eq!(packet.len() - start, packet_size); self.active_query_sent_settings = !query.settings.is_empty(); self.active_query_has_server_packet = false; @@ -1511,6 +1518,13 @@ impl Connection { /// [`InsertBlockError`] says which. /// - [`ProtocolError::InsertOutputAllocationFailed`] if the Data packet /// cannot be reserved in the output queue. + /// - [`ProtocolError::Compression`] if LZ4 framing fails. + /// + /// With compression enabled for the active Query, the block body is + /// written directly into the output queue as LZ4 frames. Confirmed at + /// `v26.8.1.2041-lts`, `src/Client/Connection.cpp::Connection::sendData` + /// flushes `CompressedWriteBuffer` after each Native block, keeping the + /// packet code and table name outside compression. pub fn insert_data(&mut self, native_block: &[u8]) -> Result<(), ProtocolError> { if self.state != State::Inserting { return Err(ProtocolError::InvalidState { @@ -1573,11 +1587,19 @@ impl Connection { )); } - let additional = native_block.len().checked_add(2).ok_or( - ProtocolError::InsertOutputAllocationFailed { - additional: usize::MAX, - }, - )?; + let additional = if self.active_query_compression { + native_block + .chunks(INSERT_FRAME_PLAINTEXT) + .try_fold(2_usize, |total, chunk| { + let len = chunk.len(); + total.checked_add(FRAME_PREFIX_LEN + len + len / 255 + 16) + }) + } else { + native_block.len().checked_add(2) + } + .ok_or(ProtocolError::InsertOutputAllocationFailed { + additional: usize::MAX, + })?; // Reserve the whole packet before writing any of it, so a failure // leaves `output` untouched and the header and body are written // without a reallocation between them. Exact sizing for an empty @@ -1589,9 +1611,24 @@ impl Connection { self.output.try_reserve(additional) }; reserved.map_err(|_| ProtocolError::InsertOutputAllocationFailed { additional })?; + let start = self.output.len(); put_varuint(&mut self.output, CLIENT_PACKET_DATA); put_string(&mut self.output, ""); - self.output.extend_from_slice(native_block); + if self.active_query_compression { + for chunk in native_block.chunks(INSERT_FRAME_PLAINTEXT) { + if let Err(error) = write_lz4_frame(chunk, &mut self.output) { + self.output.truncate(start); + return Err(match error { + CompressionError::AllocationFailed { .. } => { + ProtocolError::InsertOutputAllocationFailed { additional } + } + error => error.into(), + }); + } + } + } else { + self.output.extend_from_slice(native_block); + } Ok(()) } @@ -1614,7 +1651,7 @@ impl Connection { state: self.state, }); } - put_empty_data_packet(&mut self.output); + put_empty_data_packet(&mut self.output, self.active_query_compression); self.state = State::FinishingInsert; Ok(()) } @@ -1672,14 +1709,14 @@ impl Connection { /// EOF closes a `Ready` engine successfully. It is truncation during /// handshake, SELECT, INSERT, or Ping, or whenever a packet is incomplete. /// After a terminal server Exception it succeeds unless bytes followed - /// the Exception. Every INSERT Exception is treated as terminal even when - /// the server may instead be waiting in its skip-data path. Reusable - /// query-time exceptions leave the engine Ready instead (see - /// [`Event::Exception`]). After a protocol failure it returns the error - /// that failed the connection. EOF that fails the connection discards - /// untaken output, as in [`Self::feed`]. Events queued before EOF stay - /// queued; drain them afterwards as usual. The first EOF outcome is - /// stored and every later call returns it again. + /// the Exception. Reusable SELECT and INSERT exceptions leave the engine + /// Ready instead. An INSERT Exception after the schema remains terminal + /// because its Native reader may be misaligned. See + /// [`Event::Exception`] for the reuse rules. After a protocol failure it + /// returns the error that failed the connection. EOF that fails the + /// connection discards untaken output, as in [`Self::feed`]. Events queued + /// before EOF stay queued; drain them afterwards as usual. The first EOF + /// outcome is stored and every later call returns it again. pub fn finish(&mut self) -> Result<(), ProtocolError> { if let Some(result) = &self.finish_result { return result.clone(); @@ -1919,8 +1956,10 @@ impl Connection { self.state, State::AwaitingInsertSchema | State::Inserting | State::FinishingInsert ); - let query_exception = self.state == State::Querying; - let terminal = !query_exception + let reusable_phase = self.state == State::Querying + || (self.state == State::AwaitingInsertSchema + && self.output.is_empty()); + let terminal = !reusable_phase || is_terminal_query_exception( &exception, self.active_query_sent_settings, @@ -1995,7 +2034,13 @@ impl Connection { let mut cursor = Cursor::new(bytes); let packet = cursor.varuint()?; - let first_query_packet = if self.state == State::Querying { + let first_query_packet = if matches!( + self.state, + State::Querying + | State::AwaitingInsertSchema + | State::Inserting + | State::FinishingInsert + ) { let first = !self.active_query_has_server_packet; self.active_query_has_server_packet = true; first @@ -2116,10 +2161,9 @@ impl Connection { fn start_block(&mut self, kind: BlockKind) { let revision = self.effective_revision.unwrap_or(ENGINE_REVISION); let compressed = self.active_query_compression - && self.state == State::Querying && matches!( kind, - BlockKind::Data | BlockKind::Totals | BlockKind::Extremes + BlockKind::Data | BlockKind::Totals | BlockKind::Extremes | BlockKind::InsertSchema ); self.pending = Some(PendingPacket::Block(BlockProgress::new( kind, @@ -2852,10 +2896,18 @@ fn put_string(output: &mut Vec, value: &str) { /// Writes a client Data packet carrying `Block::empty()` at revision 54460. /// Confirmed at `v26.8.1.2041-lts`, `src/Server/TCPHandler.cpp`, /// `TCPHandler::processData` treats any zero-column block as end-of-upload. -fn put_empty_data_packet(output: &mut Vec) { +fn put_empty_data_packet(output: &mut Vec, compressed: bool) { put_varuint(output, CLIENT_PACKET_DATA); put_string(output, ""); - output.extend_from_slice(EMPTY_NATIVE_BLOCK); + output.extend_from_slice(empty_native_block(compressed)); +} + +fn empty_native_block(compressed: bool) -> &'static [u8] { + if compressed { + EMPTY_NATIVE_BLOCK_LZ4 + } else { + EMPTY_NATIVE_BLOCK + } } fn escaped_content_len(value: &[u8]) -> Option { diff --git a/crates/ch-tcp-proto/src/tests.rs b/crates/ch-tcp-proto/src/tests.rs index d290426..74320bf 100644 --- a/crates/ch-tcp-proto/src/tests.rs +++ b/crates/ch-tcp-proto/src/tests.rs @@ -1,5 +1,5 @@ use super::*; -use ch_core_rs::compression::{compress_lz4_frame, write_lz4_frame}; +use ch_core_rs::compression::{compress_lz4_frame, decompress_frame}; const EXCEPTION_FAILED: State = State::Failed(FailureKind::ServerException); const PROTOCOL_FAILED: State = State::Failed(FailureKind::Protocol); @@ -1118,14 +1118,14 @@ fn active_query_snapshots_compression_setting() { } #[test] -fn compressed_client_fed_insert_is_rejected_atomically() { +fn compressed_insert_queues_the_same_request_as_query() { let mut engine = ready_engine().with_compression(true); - assert_eq!( - engine.insert(Query::new("INSERT INTO items VALUES")), - Err(ProtocolError::CompressedInsertUnsupported) - ); - assert_eq!(engine.state(), State::Ready); - assert!(engine.take_output().is_empty()); + let query = Query::new("INSERT INTO items VALUES"); + let mut via_query = ready_engine().with_compression(true); + via_query.query(query.clone()).unwrap(); + engine.insert(query).unwrap(); + assert_eq!(engine.state(), State::AwaitingInsertSchema); + assert_eq!(engine.take_output(), via_query.take_output()); } #[test] @@ -1453,6 +1453,242 @@ fn inserting_engine() -> Connection { engine } +fn compressed_inserting_engine() -> Connection { + let mut engine = ready_engine().with_compression(true); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + let _ = engine.take_output(); + engine + .feed(&compressed_packet( + SERVER_PACKET_DATA, + &[&framed_int64_block(&[])], + )) + .unwrap(); + assert!(matches!( + events(&mut engine).as_slice(), + [Event::InsertSchema(_)] + )); + assert_eq!(engine.state(), State::Inserting); + engine +} + +fn take_compressed_client_block(bytes: &mut &[u8]) -> (Vec, usize) { + assert_eq!(&bytes[..2], &[CLIENT_PACKET_DATA as u8, 0]); + *bytes = &bytes[2..]; + let mut native = Vec::new(); + let mut frames = 0; + let mut scanner = native_scanner(ENGINE_REVISION); + loop { + let frame = decompress_frame(bytes, ChecksumMode::Verify).unwrap(); + assert!(frame.plaintext_len() <= INSERT_FRAME_PLAINTEXT); + native.extend_from_slice(frame.plaintext()); + let consumed = frame.consumed(); + *bytes = &bytes[consumed..]; + frames += 1; + if let BlockBoundary::Complete { consumed, .. } = scanner.scan(&native).unwrap() { + assert_eq!(consumed, native.len()); + return (native, frames); + } + } +} + +#[test] +fn compressed_insert_schema_and_raw_metadata_are_split_independent() { + let schema = framed_int64_block(&[]); + let telemetry = framed_int64_block(&[79]); + let split = schema.len() / 2; + let mut response = table_columns_packet("", "columns format version: 1\n"); + response.extend(compressed_packet( + SERVER_PACKET_DATA, + &[&schema[..split], &schema[split..]], + )); + for packet in [SERVER_PACKET_LOG, SERVER_PACKET_PROFILE_EVENTS] { + response.extend([packet as u8, 0]); + response.extend_from_slice(&telemetry); + } + for chunk in 1..=response.len() { + let mut engine = ready_engine().with_compression(true); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + let _ = engine.take_output(); + for part in response.chunks(chunk) { + engine.feed(part).unwrap(); + } + assert_eq!(engine.state(), State::Inserting, "chunk {chunk}"); + assert_eq!( + events(&mut engine), + [ + Event::TableColumns { + table_name: Vec::new(), + description: b"columns format version: 1\n".to_vec(), + truncated: false, + }, + Event::InsertSchema(schema.clone()), + Event::Log(telemetry.clone()), + Event::ProfileEvents(telemetry.clone()), + ] + ); + } +} + +#[test] +fn compressed_insert_writes_multiple_blocks_and_frames() { + let blocks = [ + framed_int64_block(&[13, 79]), + framed_int64_block(&[]), + framed_string_block(INSERT_FRAME_PLAINTEXT + 13), + ]; + let mut engine = compressed_inserting_engine(); + for block in &blocks { + engine.insert_data(block).unwrap(); + } + engine.finish_insert().unwrap(); + let output = engine.take_output(); + let mut remaining = output.as_slice(); + for (index, block) in blocks.iter().enumerate() { + let (decoded, frames) = take_compressed_client_block(&mut remaining); + assert_eq!(&decoded, block); + assert_eq!(frames, if index == 2 { 2 } else { 1 }); + } + assert_eq!( + take_compressed_client_block(&mut remaining), + (EMPTY_NATIVE_BLOCK.to_vec(), 1) + ); + assert!(remaining.is_empty()); + engine.feed(&[SERVER_PACKET_END_OF_STREAM as u8]).unwrap(); + assert_eq!(events(&mut engine), [Event::EndOfStream]); + assert_eq!(engine.state(), State::Ready); +} + +#[test] +fn insert_snapshots_compression_for_schema_upload_and_finish() { + let block = framed_int64_block(&[13, 79]); + for compressed in [false, true] { + let mut engine = ready_engine().with_compression(compressed); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + let _ = engine.take_output(); + engine = engine.with_compression(!compressed); + let schema = framed_int64_block(&[]); + let packet = if compressed { + compressed_packet(SERVER_PACKET_DATA, &[&schema]) + } else { + data_packet(&schema) + }; + engine.feed(&packet).unwrap(); + assert_eq!(events(&mut engine), [Event::InsertSchema(schema)]); + engine.insert_data(&block).unwrap(); + engine.finish_insert().unwrap(); + let output = engine.take_output(); + if compressed { + let mut remaining = output.as_slice(); + assert_eq!(take_compressed_client_block(&mut remaining).0, block); + assert_eq!( + take_compressed_client_block(&mut remaining).0, + EMPTY_NATIVE_BLOCK + ); + assert!(remaining.is_empty()); + } else { + let mut expected = vec![CLIENT_PACKET_DATA as u8, 0]; + expected.extend_from_slice(&block); + put_empty_data_packet(&mut expected, false); + assert_eq!(output, expected); + } + } +} + +#[test] +fn compressed_insert_rejects_invalid_uploads_atomically() { + let block = framed_string_block_of_len(MIN_MAX_BLOCK_BYTES, |_| b'x'); + let mut engine = compressed_inserting_engine().with_max_block_bytes(block.len()); + engine.insert_data(&block).unwrap(); + let before = engine.output.clone(); + let oversized = framed_string_block(MIN_MAX_BLOCK_BYTES + 13); + let concatenated = [framed_int64_block(&[13]), framed_int64_block(&[79])].concat(); + for invalid in [ + oversized.as_slice(), + &[0xff; 16], + &block[..block.len() - 1], + concatenated.as_slice(), + EMPTY_NATIVE_BLOCK, + ] { + assert!(matches!( + engine.insert_data(invalid), + Err(ProtocolError::InvalidInsertBlock(_)) + )); + assert_eq!(engine.output, before); + assert_eq!(engine.state(), State::Inserting); + } + let zero_rows = framed_int64_block(&[]); + engine.insert_data(&zero_rows).unwrap(); + let output = engine.take_output(); + let mut remaining = output.as_slice(); + assert_eq!(take_compressed_client_block(&mut remaining).0, block); + assert_eq!(take_compressed_client_block(&mut remaining).0, zero_rows); + assert!(remaining.is_empty()); +} + +#[test] +fn compressed_insert_schema_rejects_malformed_or_oversized_frames() { + let schema = framed_int64_block(&[]); + let mut checksum = compressed_packet(SERVER_PACKET_DATA, &[&schema]); + checksum[2] ^= 1; + let mut codec = compressed_packet(SERVER_PACKET_DATA, &[&schema]); + codec[2 + 16] = 0x7f; + let mut trailing = schema.clone(); + trailing.push(0); + let mut oversized = vec![SERVER_PACKET_DATA as u8, 0]; + oversized.extend(none_frame(&vec![0; MIN_MAX_BLOCK_BYTES + 1])); + for packet in [ + checksum, + codec, + compressed_packet(SERVER_PACKET_DATA, &[&trailing]), + oversized, + ] { + for chunk in [1, packet.len()] { + let mut engine = ready_engine() + .with_compression(true) + .with_max_block_bytes(MIN_MAX_BLOCK_BYTES); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + let mut error = None; + for part in packet.chunks(chunk) { + if let Err(failure) = engine.feed(part) { + error = Some(failure); + break; + } + } + assert!(matches!( + error, + Some(ProtocolError::Compression(_)) + | Some(ProtocolError::NativeBlockTooLarge { .. }) + )); + assert_eq!(engine.state(), PROTOCOL_FAILED); + assert!(engine.take_output().is_empty()); + assert!(events(&mut engine).is_empty()); + } + } +} + +#[test] +fn compressed_insert_truncated_schema_reports_transport_eof() { + let packet = compressed_packet(SERVER_PACKET_DATA, &[&framed_int64_block(&[])]); + for end in 0..packet.len() { + let mut engine = ready_engine().with_compression(true); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + engine.feed(&packet[..end]).unwrap(); + assert_eq!(engine.finish(), Err(ProtocolError::UnexpectedEof)); + assert_eq!(engine.state(), PROTOCOL_FAILED); + assert!(engine.take_output().is_empty()); + } +} + #[test] fn insert_queues_query_and_external_table_terminator() { let mut engine = ready_engine(); @@ -1463,7 +1699,7 @@ fn insert_queues_query_and_external_table_terminator() { let output = engine.take_output(); assert_eq!(output.first(), Some(&(CLIENT_PACKET_QUERY as u8))); let mut terminator = Vec::new(); - put_empty_data_packet(&mut terminator); + put_empty_data_packet(&mut terminator, false); assert!(output.ends_with(&terminator)); } @@ -1849,7 +2085,7 @@ fn insert_finish_is_single_use_and_waits_for_end_of_stream() { engine.finish_insert().unwrap(); assert_eq!(engine.state(), State::FinishingInsert); let mut terminator = Vec::new(); - put_empty_data_packet(&mut terminator); + put_empty_data_packet(&mut terminator, false); assert_eq!(engine.take_output(), terminator); assert!(matches!( engine.finish_insert(), @@ -1891,7 +2127,7 @@ fn insert_rejects_end_of_stream_table_columns_and_data_during_upload() { } #[test] -fn insert_exception_in_every_phase_fails_and_discards_output() { +fn insert_exception_with_queued_or_unfinished_upload_fails_and_discards_output() { let failure = exception(241, b"DB::Exception", b"insert failed", b"", 0); let mut awaiting = ready_engine(); @@ -1925,6 +2161,186 @@ fn insert_exception_in_every_phase_fails_and_discards_output() { } } +#[test] +fn insert_exception_reuse_requires_transferred_request_before_schema() { + let failure = exception(241, b"DB::Exception", b"insert failed", b"", 0); + for compressed in [false, true] { + for phase in [ + State::AwaitingInsertSchema, + State::Inserting, + State::FinishingInsert, + ] { + for transfer_output in [false, true] { + for split in 0..=failure.len() { + let mut engine = if phase == State::AwaitingInsertSchema { + let mut engine = ready_engine().with_compression(compressed); + engine + .insert(Query::new("INSERT INTO missing VALUES")) + .unwrap(); + engine + } else if compressed { + compressed_inserting_engine() + } else { + inserting_engine() + }; + if phase != State::AwaitingInsertSchema { + engine.insert_data(&framed_int64_block(&[13])).unwrap(); + } + if phase == State::FinishingInsert { + engine.finish_insert().unwrap(); + } + if transfer_output { + assert!(!engine.take_output().is_empty()); + } + engine.feed(&failure[..split]).unwrap(); + engine.feed(&failure[split..]).unwrap(); + let reusable = transfer_output && phase == State::AwaitingInsertSchema; + assert_eq!( + engine.state(), + if reusable { + State::Ready + } else { + EXCEPTION_FAILED + }, + "compressed {compressed}, phase {phase:?}, transfer {transfer_output}, split {split}" + ); + assert!(engine.take_output().is_empty()); + assert!(matches!( + events(&mut engine).as_slice(), + [Event::Exception(error)] if error.code == 241 + )); + if reusable { + engine.query(Query::new("SELECT 79")).unwrap(); + let block = framed_int64_block(&[79]); + let mut response = if compressed { + compressed_packet(SERVER_PACKET_DATA, &[&block]) + } else { + data_packet(&block) + }; + response.push(SERVER_PACKET_END_OF_STREAM as u8); + engine.feed(&response).unwrap(); + assert_eq!( + events(&mut engine), + [Event::Data(block), Event::EndOfStream] + ); + assert_eq!(engine.state(), State::Ready); + } + } + } + } + } +} + +#[test] +fn insert_cast_exception_after_transferred_terminator_is_terminal() { + let block = |first_type: &str, rows: u64, first: &[u8], second: &[u8]| { + let mut bytes = vec![1, 0, 2]; + bytes.extend_from_slice(&(-1_i32).to_le_bytes()); + bytes.push(0); + put_varuint(&mut bytes, 2); + put_varuint(&mut bytes, rows); + for (name, type_name, data) in [("a", first_type, first), ("c", "String", second)] { + put_string(&mut bytes, name); + put_string(&mut bytes, type_name); + bytes.push(0); + bytes.extend_from_slice(data); + } + bytes + }; + let schema = block("UInt64", 0, &[], &[]); + let upload = block("String", 1, b"\x03abc", b"\x06unread"); + let failure = exception( + 6, + b"DB::Exception", + b"Cannot parse string 'abc' as UInt64", + b"", + 0, + ); + for compressed in [false, true] { + for split in 0..=failure.len() { + let mut engine = ready_engine().with_compression(compressed); + engine + .insert(Query::new("INSERT INTO items VALUES")) + .unwrap(); + let _ = engine.take_output(); + let packet = if compressed { + compressed_packet(SERVER_PACKET_DATA, &[&schema]) + } else { + data_packet(&schema) + }; + engine.feed(&packet).unwrap(); + assert_eq!(events(&mut engine), [Event::InsertSchema(schema.clone())]); + engine.insert_data(&upload).unwrap(); + engine.finish_insert().unwrap(); + assert!(!engine.take_output().is_empty()); + engine.feed(&failure[..split]).unwrap(); + engine.feed(&failure[split..]).unwrap(); + assert_eq!(engine.state(), EXCEPTION_FAILED); + assert!(matches!( + events(&mut engine).as_slice(), + [Event::Exception(error)] if error.code == 6 + )); + assert!(matches!( + engine.query(Query::new("SELECT 79")), + Err(ProtocolError::InvalidState { .. }) + )); + } + } +} + +#[test] +fn compressed_insert_reserves_the_full_packet_bound_before_writing() { + let mut engine = compressed_inserting_engine(); + let block = framed_string_block(INSERT_FRAME_PLAINTEXT + 13); + let bound = 2 + block + .chunks(INSERT_FRAME_PLAINTEXT) + .map(|chunk| FRAME_PREFIX_LEN + chunk.len() + chunk.len() / 255 + 16) + .sum::(); + engine.insert_data(&block).unwrap(); + assert!(engine.output.capacity() >= bound); + engine.output.clear(); + let before = engine.output.as_ptr(); + engine.insert_data(&block).unwrap(); + assert_eq!(engine.output.as_ptr(), before); + let output = engine.take_output(); + let mut bytes = output.as_slice(); + assert_eq!(take_compressed_client_block(&mut bytes), (block, 2)); + assert!(bytes.is_empty()); +} + +#[test] +fn insert_exception_reuse_preserves_terminal_codes_and_wire_settings_rules() { + for compressed in [false, true] { + for (code, settings, prior_packet, expected) in [ + (62, false, false, State::Ready), + (62, true, false, EXCEPTION_FAILED), + (62, true, true, State::Ready), + (702, false, true, EXCEPTION_FAILED), + (101, false, false, EXCEPTION_FAILED), + ] { + let mut engine = ready_engine().with_compression(compressed); + let mut request = Query::new("INSERT INTO missing VALUES"); + if settings { + request = request.with_setting("max_threads", "1"); + } + engine.insert(request).unwrap(); + let _ = engine.take_output(); + if prior_packet { + engine.feed(&table_columns_packet("", "columns")).unwrap(); + let _ = events(&mut engine); + } + engine + .feed(&exception(code, b"DB::Exception", b"failed", b"", 0)) + .unwrap(); + assert_eq!( + engine.state(), + expected, + "compressed {compressed}, code {code}, settings {settings}, prior {prior_packet}" + ); + } + } +} + #[test] fn insert_rejects_phase_errors_and_truncated_schema() { let mut ready = ready_engine(); diff --git a/crates/ch-tcp-proto/tests/compressed_insert_replay.rs b/crates/ch-tcp-proto/tests/compressed_insert_replay.rs new file mode 100644 index 0000000..7b24aaf --- /dev/null +++ b/crates/ch-tcp-proto/tests/compressed_insert_replay.rs @@ -0,0 +1,457 @@ +//! Replay compressed INSERT captures without sockets, plus an opt-in live check. + +use ch_core_rs::batch::ColBatch; +use ch_core_rs::column::{Column, PrimitiveColumn, Utf8Column}; +use ch_core_rs::compression::{decompress_native_stream, ChecksumMode}; +use ch_core_rs::native::decode::{decode_all_bytes, DecodeOptions}; +use ch_core_rs::native::encode::{encode_block, EncodeOptions}; +use ch_core_rs::schema::{ChType, Field, Schema}; +use ch_tcp_proto::{ClientHello, Connection, Event, FailureKind, Query, State}; + +const HELLO: &[u8] = + include_bytes!("../../../tests/fixtures/tcp_handshake_server_hello_rev54460.packet"); +const TABLE: &str = "ch_core_tcp_compressed_insert_fixture"; + +struct Capture { + label: &'static str, + request: &'static [u8], + schema: &'static [u8], + blocks: &'static [&'static [u8]], + finish: &'static [u8], + completion: &'static [u8], + recovery_request: &'static [u8], + recovery: &'static [u8], +} + +macro_rules! fixture { + ($label:literal, $stage:literal) => { + include_bytes!(concat!( + "../../../tests/fixtures/tcp_compressed_insert_", + $label, + "_", + $stage, + "_rev54460.bytes" + )) + }; +} + +macro_rules! capture { + ($label:literal, [$($block:literal),*]) => { + Capture { + label: $label, + request: fixture!($label, "request"), + schema: fixture!($label, "schema"), + blocks: &[$(fixture!($label, $block)),*], + finish: fixture!($label, "finish"), + completion: fixture!($label, "completion"), + recovery_request: fixture!($label, "recovery_request"), + recovery: fixture!($label, "recovery"), + } + }; +} + +const CAPTURES: &[Capture] = &[ + capture!("multi", ["block0", "block1"]), + Capture { + label: "rejected", + request: fixture!("rejected", "request"), + schema: fixture!("rejected", "schema"), + blocks: &[fixture!("rejected", "block0")], + finish: fixture!("rejected", "finish"), + completion: fixture!("rejected", "completion"), + recovery_request: &[], + recovery: &[], + }, + Capture { + label: "before_schema", + request: fixture!("before_schema", "request"), + schema: &[], + blocks: &[], + finish: &[], + completion: fixture!("before_schema", "completion"), + recovery_request: fixture!("before_schema", "recovery_request"), + recovery: fixture!("before_schema", "recovery"), + }, + Capture { + label: "plain_before_schema", + request: fixture!("plain_before_schema", "request"), + schema: &[], + blocks: &[], + finish: &[], + completion: fixture!("plain_before_schema", "completion"), + recovery_request: fixture!("plain_before_schema", "recovery_request"), + recovery: fixture!("plain_before_schema", "recovery"), + }, +]; + +fn hello() -> ClientHello { + ClientHello::new("ch-core-query-fixture", "default") + .with_client_version(13, 79) + .with_quota_key("fixture_quota_13") +} + +fn ready() -> Connection { + let mut connection = Connection::new(hello()).unwrap().with_compression(true); + connection.take_output(); + connection.feed(HELLO).unwrap(); + assert!(matches!( + connection.poll_event(), + Some(Event::ServerHello(_)) + )); + assert_eq!(connection.take_output(), b"\x10fixture_quota_13"); + connection +} + +fn query(sql: String, label: &str, settings: bool) -> Query { + let query = Query::new(sql) + .with_query_id(format!("ch-core-compressed-insert-{label}")) + .with_initial_address("192.0.2.13:79") + .with_initial_query_start_time_microseconds(1_791_500_000_000_013) + .with_os_user("fixture_os_user_13") + .with_hostname("fixture_host_79"); + if settings { + query + .with_setting("send_profile_events", "0") + .with_setting("send_logs_level", "none") + } else { + query + } +} + +fn feed(connection: &mut Connection, bytes: &[u8], chunk: usize) -> Vec { + let mut events = Vec::new(); + for bytes in bytes.chunks(chunk.max(1)) { + connection.feed(bytes).unwrap(); + events.extend(std::iter::from_fn(|| connection.poll_event())); + } + events +} + +fn native(packet: &[u8]) -> Vec { + assert_eq!(&packet[..2], &[2, 0]); + decompress_native_stream(&packet[2..], ChecksumMode::Verify).unwrap() +} + +fn batch(rows: &[(u64, &str)]) -> ColBatch { + let mut offsets = vec![0]; + let mut data = Vec::new(); + for (_, value) in rows { + data.extend_from_slice(value.as_bytes()); + offsets.push(data.len() as i32); + } + ColBatch::new( + Schema::new(vec![ + Field { + name: "n".into(), + ch_type: ChType::UInt64, + }, + Field { + name: "s".into(), + ch_type: ChType::String, + }, + ]), + vec![ + Column::UInt64(PrimitiveColumn::new( + rows.iter().map(|(number, _)| *number).collect(), + )), + Column::Utf8(Utf8Column::new(offsets, data)), + ], + rows.len(), + ) +} + +fn encode(batch: &ColBatch) -> Vec { + let mut options = EncodeOptions::default(); + options.protocol_revision = 54_460; + encode_block(batch, &options).unwrap() +} + +fn start(capture: &Capture) -> Connection { + let mut connection = ready().with_compression(capture.label != "plain_before_schema"); + let before_schema = capture.schema.is_empty(); + let sql = if before_schema { + "INSERT INTO ch_core_tcp_compressed_insert_missing (n, s) VALUES".to_owned() + } else { + format!("INSERT INTO {TABLE} (n, s) VALUES") + }; + connection + .insert(query(sql, capture.label, !before_schema)) + .unwrap(); + assert_eq!(connection.take_output(), capture.request); + connection +} + +fn upload(connection: &mut Connection, capture: &Capture) { + let large = "x".repeat(1_100_000); + for (index, captured) in capture.blocks.iter().enumerate() { + let rows = match (capture.label, index) { + ("multi", 0) => vec![(13, "user_1")], + ("multi", 1) => vec![(79, large.as_str())], + ("rejected", 0) => vec![(179, "rejected")], + _ => panic!("unexpected capture block"), + }; + let encoded = encode(&batch(&rows)); + assert_eq!(encoded, native(captured)); + connection.insert_data(&encoded).unwrap(); + assert_eq!(connection.take_output(), *captured); + } + connection.finish_insert().unwrap(); + assert_eq!(connection.take_output(), capture.finish); +} + +fn assert_schema(events: &[Event]) { + assert!(events.iter().any(|event| matches!( + event, + Event::TableColumns { + truncated: false, + .. + } + ))); + let native = events + .iter() + .find_map(|event| match event { + Event::InsertSchema(bytes) => Some(bytes.as_slice()), + _ => None, + }) + .unwrap(); + assert_eq!(native, encode(&batch(&[]))); +} + +fn recovery(connection: &mut Connection, capture: &Capture, chunk: usize) -> Vec { + let sql = if capture.schema.is_empty() { + "SELECT toUInt64(13) AS n, 'user_1' AS s".to_owned() + } else { + format!("SELECT n, s FROM {TABLE} ORDER BY n") + }; + connection + .query(query(sql, &format!("{}_recovery", capture.label), true)) + .unwrap(); + assert_eq!(connection.take_output(), capture.recovery_request); + let events = feed(connection, capture.recovery, chunk); + assert!(matches!(events.last(), Some(Event::EndOfStream))); + assert_eq!(connection.state(), State::Ready); + let native: Vec = events + .iter() + .filter_map(|event| match event { + Event::Data(bytes) => Some(bytes.as_slice()), + _ => None, + }) + .flatten() + .copied() + .collect(); + let mut options = DecodeOptions::default(); + options.protocol_revision = 54_460; + let decoded = decode_all_bytes(&native, &options).unwrap(); + if capture.label == "multi" { + let large = "x".repeat(1_100_000); + assert_eq!(decoded.num_rows(), 2); + assert_eq!(decoded.chunks.len(), 1); + assert_eq!( + *decoded.chunks[0], + batch(&[(13, "user_1"), (79, large.as_str())]) + ); + } else { + assert_eq!(decoded.num_rows(), 1); + assert_eq!(*decoded.chunks[0], batch(&[(13, "user_1")])); + } + events +} + +#[test] +fn captured_compressed_insert_and_exception_recovery_replay_fragmented() { + for capture in CAPTURES { + let mut expected = None; + for chunk in [usize::MAX, 1, 2, 7, 31, 127] { + let mut connection = start(capture); + let schema = feed(&mut connection, capture.schema, chunk); + if !capture.schema.is_empty() { + assert_schema(&schema); + upload(&mut connection, capture); + } + let completion = feed(&mut connection, capture.completion, chunk); + if capture.label == "rejected" { + let Some(Event::Exception(exception)) = completion.last() else { + panic!("missing rejection"); + }; + assert_eq!(exception.name, "DB::Exception"); + assert!(exception.message.contains("Constraint")); + assert!(!completion + .iter() + .any(|event| matches!(event, Event::EndOfStream))); + assert_eq!( + connection.state(), + State::Failed(FailureKind::ServerException) + ); + } else if capture.schema.is_empty() { + assert!(matches!( + completion.last(), + Some(Event::Exception(exception)) if exception.code == 60 + )); + assert_eq!(connection.state(), State::Ready); + } else { + assert!(matches!(completion.last(), Some(Event::EndOfStream))); + assert_eq!(connection.state(), State::Ready); + } + let recovered = if capture.recovery.is_empty() { + Vec::new() + } else { + recovery(&mut connection, capture, chunk) + }; + let actual = (schema, completion, recovered); + if let Some(expected) = &expected { + assert_eq!(&actual, expected, "{} chunk {chunk}", capture.label); + } else { + expected = Some(actual); + } + } + } +} + +#[test] +fn captured_schema_and_completion_replay_at_every_split() { + for capture in CAPTURES { + for stage in ["schema", "completion"] { + if stage == "schema" && capture.schema.is_empty() { + continue; + } + let bytes = if stage == "schema" { + capture.schema + } else { + capture.completion + }; + for split in 0..=bytes.len() { + let mut connection = start(capture); + if stage == "completion" && !capture.schema.is_empty() { + assert_schema(&feed(&mut connection, capture.schema, usize::MAX)); + upload(&mut connection, capture); + } + let mut events = feed(&mut connection, &bytes[..split], usize::MAX); + events.extend(feed(&mut connection, &bytes[split..], usize::MAX)); + if stage == "schema" { + assert_schema(&events); + assert_eq!(connection.state(), State::Inserting); + } else { + if capture.label == "rejected" { + assert_eq!( + connection.state(), + State::Failed(FailureKind::ServerException) + ); + } else { + assert_eq!(connection.state(), State::Ready); + recovery(&mut connection, capture, usize::MAX); + } + } + } + } + } +} + +#[test] +fn upload_phase_exception_retires_compressed_connection() { + let capture = &CAPTURES[1]; + let mut connection = start(capture); + assert_schema(&feed(&mut connection, capture.schema, 1)); + connection + .insert_data(&encode(&batch(&[(179, "rejected")]))) + .unwrap(); + let events = feed(&mut connection, capture.completion, 1); + assert!(matches!(events.last(), Some(Event::Exception(_)))); + assert_eq!( + connection.state(), + State::Failed(FailureKind::ServerException) + ); + assert!(connection.take_output().is_empty()); +} + +#[test] +#[ignore = "requires ClickHouse v26.8.1.2041-lts on 127.0.0.1:9000"] +fn live_compressed_insert_select_round_trip() { + use std::io::{Read, Write}; + use std::net::TcpStream; + use std::time::Duration; + + fn receive( + socket: &mut TcpStream, + connection: &mut Connection, + done: impl Fn(&Event) -> bool, + ) -> Vec { + let mut events = Vec::new(); + let mut buffer = [0; 4096]; + loop { + let count = socket.read(&mut buffer).unwrap(); + assert_ne!(count, 0, "unexpected server EOF"); + connection.feed(&buffer[..count]).unwrap(); + let mut complete = false; + while let Some(event) = connection.poll_event() { + assert!(!matches!(event, Event::Exception(_)), "{event:?}"); + complete |= done(&event); + events.push(event); + } + if complete { + return events; + } + } + } + + fn run(socket: &mut TcpStream, connection: &mut Connection, sql: String) -> Vec { + connection.query(Query::new(sql)).unwrap(); + socket.write_all(&connection.take_output()).unwrap(); + receive(socket, connection, |event| { + matches!(event, Event::EndOfStream) + }) + } + + let mut socket = TcpStream::connect("127.0.0.1:9000").unwrap(); + socket + .set_read_timeout(Some(Duration::from_secs(15))) + .unwrap(); + socket + .set_write_timeout(Some(Duration::from_secs(15))) + .unwrap(); + let mut connection = Connection::new(hello()).unwrap().with_compression(true); + socket.write_all(&connection.take_output()).unwrap(); + receive(&mut socket, &mut connection, |event| { + matches!(event, Event::ServerHello(_)) + }); + socket.write_all(&connection.take_output()).unwrap(); + let table = format!("ch_core_tcp_compressed_live_{}", std::process::id()); + run( + &mut socket, + &mut connection, + format!("CREATE TABLE {table} (n UInt64, s String) ENGINE=Memory"), + ); + let expected = batch(&[(13, "user_1\0"), (79, "multibyte_\u{1f980}")]); + connection + .insert(Query::new(format!("INSERT INTO {table} VALUES"))) + .unwrap(); + socket.write_all(&connection.take_output()).unwrap(); + receive(&mut socket, &mut connection, |event| { + matches!(event, Event::InsertSchema(_)) + }); + connection.insert_data(&encode(&expected)).unwrap(); + connection.finish_insert().unwrap(); + socket.write_all(&connection.take_output()).unwrap(); + receive(&mut socket, &mut connection, |event| { + matches!(event, Event::EndOfStream) + }); + let events = run( + &mut socket, + &mut connection, + format!("SELECT n, s FROM {table} ORDER BY n"), + ); + run(&mut socket, &mut connection, format!("DROP TABLE {table}")); + let bytes: Vec = events + .iter() + .filter_map(|event| match event { + Event::Data(bytes) => Some(bytes.as_slice()), + _ => None, + }) + .flatten() + .copied() + .collect(); + let mut options = DecodeOptions::default(); + options.protocol_revision = 54_460; + let actual = decode_all_bytes(&bytes, &options).unwrap(); + assert_eq!(actual.chunks.len(), 1); + assert_eq!(*actual.chunks[0], expected); +} diff --git a/crates/ch-tcp-proto/tests/insert_replay.rs b/crates/ch-tcp-proto/tests/insert_replay.rs index 796e73c..51c6e99 100644 --- a/crates/ch-tcp-proto/tests/insert_replay.rs +++ b/crates/ch-tcp-proto/tests/insert_replay.rs @@ -286,10 +286,7 @@ fn replay( let oracle = ORACLES.iter().find(|o| o.label == capture.label).unwrap(); assert_oracle(capture.schema, &schema, oracle.schema); if capture.label == "before_schema" { - assert_eq!( - connection.state(), - State::Failed(FailureKind::ServerException) - ); + assert_eq!(connection.state(), State::Ready); return (schema, Vec::new(), Vec::new()); } if matches!( @@ -345,7 +342,7 @@ fn replay( State::Ready } ); - if capture.label == "completion_exception" && capture.finish.is_empty() { + if capture.label == "completion_exception" { assert!(connection.take_output().is_empty()); } assert_eq!(connection.finish(), Ok(())); diff --git a/scripts/gen_tcp_compressed_insert_fixtures.py b/scripts/gen_tcp_compressed_insert_fixtures.py new file mode 100644 index 0000000..584f6e0 --- /dev/null +++ b/scripts/gen_tcp_compressed_insert_fixtures.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +"""Capture compressed revision-54460 INSERT and same-socket SELECT recovery.""" + +import argparse +from datetime import date +import hashlib +import io +import json +import os +from pathlib import Path +import socket +import struct +import subprocess +import sys +import tempfile +import urllib.parse +import urllib.request + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import gen_tcp_query_fixtures as query +import gen_tcp_compressed_query_fixtures as compressed +from gen_tcp_fixtures import Reader, string, varuint +from gen_tcp_handshake_fixtures import read_hello + +TABLE = "ch_core_tcp_compressed_insert_fixture" +SETTINGS = [("send_profile_events", "0"), ("send_logs_level", "none")] +FINISH = varuint(2) + string("") + query.EMPTY_COMPRESSED_LZ4 +LARGE_STRING = "x" * 1_100_000 +CASES = ("multi", "rejected", "before_schema", "plain_before_schema") + + +def request(sql, label): + before_schema = "before_schema" in label + return query.query_packet( + sql, f"ch-core-compressed-insert-{label}", + settings=() if before_schema and not label.endswith("_recovery") else SETTINGS, + compression=not label.startswith("plain_"), + ) + + +def block(reader, compression): + if compression: + result = compressed.compressed_native_block(reader, "LZ4") + native = result.pop("native") + result.pop("_column_bodies") + else: + rows, columns, schema, _, native = query.native_block(reader) + result = {"rows": rows, "columns": columns, "schema": schema, "frames": []} + result.update( + native_length=len(native), + native_sha256=hashlib.sha256(native).hexdigest(), + ) + return result + + +def client_block(data, compression): + reader = Reader(io.BytesIO(data)) + assert reader.uint() == 2 and reader.string() == b"" + result = block(reader, compression) + assert not reader.source.read(1) + return result + + +def response(reader, compression=True, schema_only=False): + packets, boundaries = bytearray(), [] + while True: + reader.record.clear() + code = reader.uint() + entry = {"code": code, "offset": len(packets)} + if code == 1: + entry["name"] = reader.string().decode() + entry.update(block(reader, compression)) + elif code == 11: + entry["table_hex"] = reader.string().hex() + entry["description_hex"] = reader.string().hex() + elif code in (10, 14): + entry["name"] = reader.string().decode() + rows, columns, schema, _, native = query.native_block(reader) + entry.update(rows=rows, columns=columns, schema=schema, + native_length=len(native), + native_sha256=hashlib.sha256(native).hexdigest()) + elif code == 3: + entry["progress"] = [reader.uint() for _ in range(6)] + elif code == 6: + entry["profile"] = [reader.uint() for _ in range(3)] + entry["applied_limit"] = reader.read(1)[0] + entry["rows_before_limit"] = reader.uint() + entry["obsolete"] = reader.read(1)[0] + elif code == 2: + entry["exception_code"] = struct.unpack(" 1024 * 1024 + assert len(oracle["frames"]) > 1 + case["blocks"].append(save(label, f"block{index}", body)) + case["block_oracles"].append(oracle) + sock.sendall(body) + finish = client(label, "finish") + assert finish == FINISH + sock.sendall(finish) + case["finish"] = save(label, "finish", finish) + case["exception_phase"] = ( + "after_upload_terminator_sent" if label == "rejected" else None + ) + data, packets = response(reader, compression) + assert packets[-1]["code"] == (2 if label == "rejected" else 5) + case["completion"] = save(label, "completion", data) + case["completion_packets"] = packets + if label != "rejected": + recovery_sql = ( + "SELECT toUInt64(13) AS n, 'user_1' AS s" + if before_schema else f"SELECT n, s FROM {TABLE} ORDER BY n" + ) + recovery_request = client(label, "recovery_request") + assert recovery_request == request(recovery_sql, label + "_recovery") + sock.sendall(recovery_request) + data, packets = response(reader, compression) + assert packets[-1]["code"] == 5, packets + case["recovery_sql"] = recovery_sql + case["recovery_request"] = save(label, "recovery_request", recovery_request) + case["recovery"] = save(label, "recovery", data) + case["recovery_packets"] = packets + rows = http(f"SELECT n, s FROM {TABLE} ORDER BY n FORMAT JSONEachRow") + expected = ( + ('{"n":13,"s":"user_1"}\n{"n":79,"s":"' + LARGE_STRING + '"}\n').encode() + if label == "multi" else b"" + ) + assert rows == expected, (label, len(rows), len(expected)) + case["verified_http_rows"] = { + "count": 2 if label == "multi" else 0, + "length": len(rows), "sha256": hashlib.sha256(rows).hexdigest(), + } + manifest["cases"][label] = case + finally: + http(f"DROP TABLE IF EXISTS {TABLE}") + args.output.mkdir(parents=True, exist_ok=True) + for name, data in files.items(): + (args.output / name).write_bytes(data) + manifest["files"][name] = { + "length": len(data), "sha256": hashlib.sha256(data).hexdigest(), + } + print(f"{name}: {len(data)} bytes") + (args.output / "tcp_compressed_insert_rev54460.json").write_text( + json.dumps(manifest, indent=2) + "\n" + ) + check(args.output) + + +if __name__ == "__main__": + main() diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 577402e..73a11d1 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -482,8 +482,9 @@ block and terminator, then feeds the suffix and completion. This covers both partial packets and complete queued Log/ProfileEvents while writing. Query and terminator bytes must match the captures. Upload blocks are constructed through core's `encode_block` and their complete Data packets must match the -accepted uploads. The schema is also decoded through core. INSERT Exceptions -conservatively fail the engine connection. +accepted uploads. The schema is also decoded through core. Upload-phase +Exceptions fail the engine connection, including completion Exceptions after +a transferred upload terminator. Before-schema errors can leave it reusable. The constraint capture's `completion_probe` and `exception_phase` fields state whether the server produced the failure before or after the upload terminator. @@ -497,5 +498,64 @@ Confirmed source at `v26.8.1.2041-lts`: `src/Formats/NativeReader.cpp::NativeReader::read` describe schema delivery, client Data framing, per-block telemetry, the zero-column upload terminator, and error drain behavior. The server-reader findings distinguish that -confirmed framing from the conservative engine policy of retiring failed -INSERT connections. +confirmed framing from the conservative engine policy of retiring connections +that fail after the schema. + +## Revision 54460 compressed INSERT capture workflow + +Captured on 2026-11-17 from ClickHouse `26.8.1.2041`, source tag +`v26.8.1.2041-lts`, on `127.0.0.1` with an empty password: + +```sh +no_proxy=127.0.0.1 python3 -B scripts/gen_tcp_compressed_insert_fixtures.py +python3 -B scripts/gen_tcp_compressed_insert_fixtures.py --check +``` + +The generator invokes the Rust `gen_compressed_insert_fixture_client` tool in +a temporary directory. The tool uses `Connection`, core's Native encoder, and +the LZ4 frame writer to generate client packets. Python sends those exact bytes +to the live server and independently parses its replies. No HTTP-generated +client frames are used. The offline check needs no server or Rust process. + +The four cases are: + +- `multi`: a compressed two-block UInt64/String upload. The second block has a + 1,100,000-byte String and spans two LZ4 frames. EndOfStream confirms acceptance. + A compressed SELECT on the same socket reads the rows back, and an independent + HTTP query verifies storage. +- `rejected`: a compressed block violates a table constraint, producing code + 469 after the client sends its upload terminator. The replay expects a failed + connection and does not attempt reuse. +- `before_schema`: a missing table produces code 60 before InsertSchema. The + same socket then runs a compressed SELECT and reads a known UInt64/String row. +- `plain_before_schema`: the same before-schema error and recovery with + compression disabled. + +Before-schema INSERT requests carry no wire settings, so the first-response +settings guard does not retire the connection. Recovery SELECTs disable logs +and profile events. The temporary Memory table is dropped after capture. +The manifest stores packet fields, frame sizes, Native lengths and SHA-256 +hashes, file lengths and hashes, and independent HTTP verification results. +Large plaintext values are represented by hashes to keep the manifest small. + +`crates/ch-tcp-proto/tests/compressed_insert_replay.rs` replays all exchanges +without sockets, including every split in schema and completion and bytewise +feeds. It asserts exact client `take_output()` bytes, including the large +multi-frame block, and checks decoded recovery rows. Its ignored live test +exercises the Rust engine directly: + +```sh +no_proxy=127.0.0.1 cargo test -p ch-tcp-proto --test compressed_insert_replay live_compressed_insert_select_round_trip -- --ignored +``` + +Confirmed at `v26.8.1.2041-lts` in +`src/Server/TCPHandler.cpp`, +`TCPHandler::{initBlockInput,processData,processUnexpectedData,receivePacketsExpectData,startInsertQuery,skipData,runImpl}`, +and `src/Client/Connection.cpp`, `Connection::{sendQuery,sendData}`: +compression applies to Native bodies in both directions, and the stock client +flushes after each Native block. Its 1 MiB writer buffer can emit several frames +per block. `src/Formats/NativeReader.cpp`, `NativeReader::read`, can throw before +consuming a block. The exception drain starts a fresh reader, so unread block +bytes can prevent it from reaching an already-sent terminator. Post-schema +Exceptions remain terminal. Before-schema reuse is inferred from the consumed +initial external-table marker and verified by the two live recovery captures. diff --git a/tests/fixtures/tcp_compressed_insert_before_schema_completion_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_before_schema_completion_rev54460.bytes new file mode 100644 index 0000000..e9999a4 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_before_schema_completion_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_before_schema_recovery_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_before_schema_recovery_request_rev54460.bytes new file mode 100644 index 0000000..d738775 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_before_schema_recovery_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_before_schema_recovery_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_before_schema_recovery_rev54460.bytes new file mode 100644 index 0000000..cce6cd7 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_before_schema_recovery_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_before_schema_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_before_schema_request_rev54460.bytes new file mode 100644 index 0000000..80e7e49 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_before_schema_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_block0_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_block0_rev54460.bytes new file mode 100644 index 0000000..878cc35 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_block0_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_block1_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_block1_rev54460.bytes new file mode 100644 index 0000000..0d3ec4a Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_block1_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_completion_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_completion_rev54460.bytes new file mode 100644 index 0000000..b0b2b1c --- /dev/null +++ b/tests/fixtures/tcp_compressed_insert_multi_completion_rev54460.bytes @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/tests/fixtures/tcp_compressed_insert_multi_finish_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_finish_rev54460.bytes new file mode 100644 index 0000000..7446165 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_finish_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_recovery_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_recovery_request_rev54460.bytes new file mode 100644 index 0000000..8712e77 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_recovery_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_recovery_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_recovery_rev54460.bytes new file mode 100644 index 0000000..70477ee Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_recovery_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_request_rev54460.bytes new file mode 100644 index 0000000..ab9eb5b Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_multi_schema_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_multi_schema_rev54460.bytes new file mode 100644 index 0000000..ac80f05 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_multi_schema_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_plain_before_schema_completion_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_plain_before_schema_completion_rev54460.bytes new file mode 100644 index 0000000..e9999a4 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_plain_before_schema_completion_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_request_rev54460.bytes new file mode 100644 index 0000000..a8d3cb7 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_rev54460.bytes new file mode 100644 index 0000000..f1ebc6d Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_plain_before_schema_recovery_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_plain_before_schema_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_plain_before_schema_request_rev54460.bytes new file mode 100644 index 0000000..abdfc46 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_plain_before_schema_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rejected_block0_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_rejected_block0_rev54460.bytes new file mode 100644 index 0000000..f8213f8 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_rejected_block0_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rejected_completion_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_rejected_completion_rev54460.bytes new file mode 100644 index 0000000..8d3c255 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_rejected_completion_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rejected_finish_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_rejected_finish_rev54460.bytes new file mode 100644 index 0000000..7446165 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_rejected_finish_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rejected_request_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_rejected_request_rev54460.bytes new file mode 100644 index 0000000..c912f5a Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_rejected_request_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rejected_schema_rev54460.bytes b/tests/fixtures/tcp_compressed_insert_rejected_schema_rev54460.bytes new file mode 100644 index 0000000..ac80f05 Binary files /dev/null and b/tests/fixtures/tcp_compressed_insert_rejected_schema_rev54460.bytes differ diff --git a/tests/fixtures/tcp_compressed_insert_rev54460.json b/tests/fixtures/tcp_compressed_insert_rev54460.json new file mode 100644 index 0000000..f885247 --- /dev/null +++ b/tests/fixtures/tcp_compressed_insert_rev54460.json @@ -0,0 +1,723 @@ +{ + "captured_on": "2026-11-17", + "server_version": "26.8.1.2041", + "source_tag": "v26.8.1.2041-lts", + "declared_revision": 54460, + "client_encoder": "ch-tcp-proto Connection and core write_lz4_frame", + "cases": { + "multi": { + "sql": "INSERT INTO ch_core_tcp_compressed_insert_fixture (n, s) VALUES", + "compression": true, + "blocks": [ + "tcp_compressed_insert_multi_block0_rev54460.bytes", + "tcp_compressed_insert_multi_block1_rev54460.bytes" + ], + "block_oracles": [ + { + "rows": 1, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 70, + "frames": [ + { + "method": 130, + "compressed_size": 54, + "uncompressed_size": 45, + "checksum": "0d3f64c1607002d3dc7114218385d76b" + } + ], + "native_length": 45, + "native_sha256": "7b111c8bb8757adb1cd64945123d79daa7d80bd63da292080242ddf6f8f73f3d" + }, + { + "rows": 1, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 4423, + "frames": [ + { + "method": 130, + "compressed_size": 4170, + "uncompressed_size": 1048576, + "checksum": "c77342b6569950d7a088226e2e327b3b" + }, + { + "method": 130, + "compressed_size": 221, + "uncompressed_size": 51465, + "checksum": "088d55522e79831b6df763eab274e3d9" + } + ], + "native_length": 1100041, + "native_sha256": "6cbf647fbe95af41327f44a863b726e191b385744fec90d24584c97cebdacdaf" + } + ], + "request": "tcp_compressed_insert_multi_request_rev54460.bytes", + "schema": "tcp_compressed_insert_multi_schema_rev54460.bytes", + "schema_packets": [ + { + "code": 11, + "offset": 0, + "table_hex": "", + "description_hex": "636f6c756d6e7320666f726d61742076657273696f6e3a20310a3220636f6c756d6e733a0a606e602055496e7436340a60736020537472696e670a", + "length": 62 + }, + { + "code": 1, + "offset": 62, + "name": "", + "rows": 0, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 57, + "frames": [ + { + "method": 130, + "compressed_size": 41, + "uncompressed_size": 30, + "checksum": "04217e7711b14b1c73d53bc33b58d36f" + } + ], + "native_length": 30, + "native_sha256": "624d5c6e30f14627e2b48f99477009116fbadd01b455aadf502b3040d726775e", + "length": 59 + } + ], + "finish": "tcp_compressed_insert_multi_finish_rev54460.bytes", + "exception_phase": null, + "completion": "tcp_compressed_insert_multi_completion_rev54460.bytes", + "completion_packets": [ + { + "code": 5, + "offset": 0, + "length": 1 + } + ], + "recovery_sql": "SELECT n, s FROM ch_core_tcp_compressed_insert_fixture ORDER BY n", + "recovery_request": "tcp_compressed_insert_multi_recovery_request_rev54460.bytes", + "recovery": "tcp_compressed_insert_multi_recovery_rev54460.bytes", + "recovery_packets": [ + { + "code": 1, + "offset": 0, + "name": "", + "rows": 0, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 57, + "frames": [ + { + "method": 130, + "compressed_size": 41, + "uncompressed_size": 30, + "checksum": "04217e7711b14b1c73d53bc33b58d36f" + } + ], + "native_length": 30, + "native_sha256": "624d5c6e30f14627e2b48f99477009116fbadd01b455aadf502b3040d726775e", + "length": 59 + }, + { + "code": 1, + "offset": 59, + "name": "", + "rows": 2, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 4436, + "frames": [ + { + "method": 130, + "compressed_size": 4183, + "uncompressed_size": 1048576, + "checksum": "4f74151148416f0e1c8056b84d304605" + }, + { + "method": 130, + "compressed_size": 221, + "uncompressed_size": 51480, + "checksum": "8f198795914cd3d7946ed8f5ca00e3b8" + } + ], + "native_length": 1100056, + "native_sha256": "226491c9227769c1b256e1e7221c4c106f5f160346a078acc04e1f4cbe46b8e6", + "length": 4438 + }, + { + "code": 6, + "offset": 4497, + "profile": [ + 2, + 1, + 2097664 + ], + "applied_limit": 0, + "rows_before_limit": 0, + "obsolete": 1, + "length": 10 + }, + { + "code": 3, + "offset": 4507, + "progress": [ + 2, + 1100038, + 2, + 0, + 0, + 3411875 + ], + "length": 12 + }, + { + "code": 1, + "offset": 4519, + "name": "", + "rows": 0, + "columns": 0, + "schema": [], + "wire_length": 36, + "frames": [ + { + "method": 130, + "compressed_size": 20, + "uncompressed_size": 10, + "checksum": "a783ac6cd55c7a7cb5ac46bddb86e214" + } + ], + "native_length": 10, + "native_sha256": "e07d2d762f3e51b34b91f72c628ffbb4f0384ec1c38dd7c35e4b6b4f238f85b4", + "length": 38 + }, + { + "code": 3, + "offset": 4557, + "progress": [ + 0, + 0, + 0, + 0, + 0, + 7292 + ], + "length": 8 + }, + { + "code": 5, + "offset": 4565, + "length": 1 + } + ], + "verified_http_rows": { + "count": 2, + "length": 1100038, + "sha256": "71efed192a89b576e9e17e455e4ffdfe9b9ccf59b83f38fb9154f5830c644718" + } + }, + "rejected": { + "sql": "INSERT INTO ch_core_tcp_compressed_insert_fixture (n, s) VALUES", + "compression": true, + "blocks": [ + "tcp_compressed_insert_rejected_block0_rev54460.bytes" + ], + "block_oracles": [ + { + "rows": 1, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 72, + "frames": [ + { + "method": 130, + "compressed_size": 56, + "uncompressed_size": 47, + "checksum": "5ee5b7704bf9733c545af8dca455823e" + } + ], + "native_length": 47, + "native_sha256": "7456d9725bebac36f53e969c4748e84b3d3747d0c5aca27b2e8ec07534e52567" + } + ], + "request": "tcp_compressed_insert_rejected_request_rev54460.bytes", + "schema": "tcp_compressed_insert_rejected_schema_rev54460.bytes", + "schema_packets": [ + { + "code": 11, + "offset": 0, + "table_hex": "", + "description_hex": "636f6c756d6e7320666f726d61742076657273696f6e3a20310a3220636f6c756d6e733a0a606e602055496e7436340a60736020537472696e670a", + "length": 62 + }, + { + "code": 1, + "offset": 62, + "name": "", + "rows": 0, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 57, + "frames": [ + { + "method": 130, + "compressed_size": 41, + "uncompressed_size": 30, + "checksum": "04217e7711b14b1c73d53bc33b58d36f" + } + ], + "native_length": 30, + "native_sha256": "624d5c6e30f14627e2b48f99477009116fbadd01b455aadf502b3040d726775e", + "length": 59 + } + ], + "finish": "tcp_compressed_insert_rejected_finish_rev54460.bytes", + "exception_phase": "after_upload_terminator_sent", + "completion": "tcp_compressed_insert_rejected_completion_rev54460.bytes", + "completion_packets": [ + { + "code": 2, + "offset": 0, + "exception_code": 469, + "exception_name": "DB::Exception", + "message": "DB::Exception: Constraint `small` for table default.ch_core_tcp_compressed_insert_fixture (36a711d9-484c-4b24-b48c-e7934e44ab61) is violated at row 1. Expression: (n < 100). Column values: n = 179", + "stack_trace": "0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000014335694\n1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000ef10e58\n2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000ef106cc\n3. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, unsigned long&&, String&&, String&) @ 0x0000000017cc62a8\n4. DB::CheckConstraintsTransform::onConsume(DB::Chunk) @ 0x000000001aeca4d0\n5. void std::__function::__policy_func::__call_func[abi:fqe220101](std::__function::__policy_storage const*) @ 0x000000001aedf044\n6. DB::runStep(std::function, std::shared_ptr&) @ 0x000000001aedee90\n7. DB::ExceptionKeepingTransform::work() @ 0x000000001aede348\n8. DB::ExecutionThreadContext::executeTask() @ 0x000000001abea5ec\n9. DB::PipelineExecutor::executeStepImpl(unsigned long, DB::WorkloadResources&&, std::atomic*) @ 0x000000001abdc028\n10. void std::__function::__policy_func::__call_func[abi:fqe220101])::$_0>(std::__function::__policy_storage const*) @ 0x000000001abdd3e8\n11. ThreadPoolImpl>::ThreadFromThreadPool::worker() @ 0x000000001455d0d0\n12. void std::__function::__policy_func::__call_func[abi:fqe220101], std::function, unsigned long, unsigned long, bool, bool)::$_0>(std::__function::__policy_storage const*) @ 0x000000001456613c\n13. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000001455aeec\n14. void* std::__thread_proxy[abi:fqe220101]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x00000000145635dc\n15. ? @ 0x00000000000803c8\n16. ? @ 0x00000000000e9edc\n", + "has_nested": false, + "length": 2462 + } + ], + "verified_http_rows": { + "count": 0, + "length": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + } + }, + "before_schema": { + "sql": "INSERT INTO ch_core_tcp_compressed_insert_missing (n, s) VALUES", + "compression": true, + "blocks": [], + "block_oracles": [], + "request": "tcp_compressed_insert_before_schema_request_rev54460.bytes", + "completion": "tcp_compressed_insert_before_schema_completion_rev54460.bytes", + "completion_packets": [ + { + "code": 2, + "offset": 0, + "exception_code": 60, + "exception_name": "DB::Exception", + "message": "DB::Exception: Table default.ch_core_tcp_compressed_insert_missing does not exist. Maybe you meant default.ch_core_tcp_compressed_insert_fixture?", + "stack_trace": "0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000014335694\n1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000ef10e58\n2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000ef106cc\n3. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&, String&&) @ 0x000000000f014b64\n4. DB::IDatabase::getTable(String const&, std::shared_ptr) const @ 0x0000000017e473ec\n5. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr, std::optional*) const @ 0x0000000018da9420\n6. DB::Context::getOrCacheStorage(DB::StorageID const&, std::function, std::shared_ptr> ()>) const @ 0x0000000018c51e70\n7. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr) const @ 0x0000000018db4cb8\n8. DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x0000000018f5ced0\n9. DB::InterpreterInsertQuery::execute() @ 0x0000000018f64494\n10. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr>&, boost::intrusive_ptr&, std::shared_ptr, std::function, DB::QueryResultDetails&) @ 0x000000001947b908\n11. DB::executeQuery(std::basic_string_view>, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000194754ec\n12. DB::TCPHandler::runImpl() @ 0x000000001aa3bbd8\n13. DB::TCPHandler::run() @ 0x000000001aa5b348\n14. Poco::Net::TCPServerConnection::start() @ 0x000000001fde8f18\n15. Poco::Net::TCPServerDispatcher::run() @ 0x000000001fde9520\n16. Poco::PooledThread::run() @ 0x000000001fdaad78\n17. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001fda8ff0\n18. ? @ 0x00000000000803c8\n19. ? @ 0x00000000000e9edc\n", + "has_nested": false, + "length": 2326 + } + ], + "recovery_sql": "SELECT toUInt64(13) AS n, 'user_1' AS s", + "recovery_request": "tcp_compressed_insert_before_schema_recovery_request_rev54460.bytes", + "recovery": "tcp_compressed_insert_before_schema_recovery_rev54460.bytes", + "recovery_packets": [ + { + "code": 1, + "offset": 0, + "name": "", + "rows": 0, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 57, + "frames": [ + { + "method": 130, + "compressed_size": 41, + "uncompressed_size": 30, + "checksum": "04217e7711b14b1c73d53bc33b58d36f" + } + ], + "native_length": 30, + "native_sha256": "624d5c6e30f14627e2b48f99477009116fbadd01b455aadf502b3040d726775e", + "length": 59 + }, + { + "code": 1, + "offset": 59, + "name": "", + "rows": 1, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "wire_length": 70, + "frames": [ + { + "method": 130, + "compressed_size": 54, + "uncompressed_size": 45, + "checksum": "0d3f64c1607002d3dc7114218385d76b" + } + ], + "native_length": 45, + "native_sha256": "7b111c8bb8757adb1cd64945123d79daa7d80bd63da292080242ddf6f8f73f3d", + "length": 72 + }, + { + "code": 6, + "offset": 131, + "profile": [ + 1, + 1, + 4624 + ], + "applied_limit": 0, + "rows_before_limit": 0, + "obsolete": 1, + "length": 8 + }, + { + "code": 3, + "offset": 139, + "progress": [ + 1, + 1, + 1, + 0, + 0, + 611625 + ], + "length": 9 + }, + { + "code": 1, + "offset": 148, + "name": "", + "rows": 0, + "columns": 0, + "schema": [], + "wire_length": 36, + "frames": [ + { + "method": 130, + "compressed_size": 20, + "uncompressed_size": 10, + "checksum": "a783ac6cd55c7a7cb5ac46bddb86e214" + } + ], + "native_length": 10, + "native_sha256": "e07d2d762f3e51b34b91f72c628ffbb4f0384ec1c38dd7c35e4b6b4f238f85b4", + "length": 38 + }, + { + "code": 3, + "offset": 186, + "progress": [ + 0, + 0, + 0, + 0, + 0, + 2833 + ], + "length": 8 + }, + { + "code": 5, + "offset": 194, + "length": 1 + } + ], + "verified_http_rows": { + "count": 0, + "length": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + } + }, + "plain_before_schema": { + "sql": "INSERT INTO ch_core_tcp_compressed_insert_missing (n, s) VALUES", + "compression": false, + "blocks": [], + "block_oracles": [], + "request": "tcp_compressed_insert_plain_before_schema_request_rev54460.bytes", + "completion": "tcp_compressed_insert_plain_before_schema_completion_rev54460.bytes", + "completion_packets": [ + { + "code": 2, + "offset": 0, + "exception_code": 60, + "exception_name": "DB::Exception", + "message": "DB::Exception: Table default.ch_core_tcp_compressed_insert_missing does not exist. Maybe you meant default.ch_core_tcp_compressed_insert_fixture?", + "stack_trace": "0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000014335694\n1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000ef10e58\n2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000ef106cc\n3. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&, String&&) @ 0x000000000f014b64\n4. DB::IDatabase::getTable(String const&, std::shared_ptr) const @ 0x0000000017e473ec\n5. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr, std::optional*) const @ 0x0000000018da9420\n6. DB::Context::getOrCacheStorage(DB::StorageID const&, std::function, std::shared_ptr> ()>) const @ 0x0000000018c51e70\n7. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr) const @ 0x0000000018db4cb8\n8. DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x0000000018f5ced0\n9. DB::InterpreterInsertQuery::execute() @ 0x0000000018f64494\n10. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr>&, boost::intrusive_ptr&, std::shared_ptr, std::function, DB::QueryResultDetails&) @ 0x000000001947b908\n11. DB::executeQuery(std::basic_string_view>, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000194754ec\n12. DB::TCPHandler::runImpl() @ 0x000000001aa3bbd8\n13. DB::TCPHandler::run() @ 0x000000001aa5b348\n14. Poco::Net::TCPServerConnection::start() @ 0x000000001fde8f18\n15. Poco::Net::TCPServerDispatcher::run() @ 0x000000001fde9520\n16. Poco::PooledThread::run() @ 0x000000001fdaad78\n17. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001fda8ff0\n18. ? @ 0x00000000000803c8\n19. ? @ 0x00000000000e9edc\n", + "has_nested": false, + "length": 2326 + } + ], + "recovery_sql": "SELECT toUInt64(13) AS n, 'user_1' AS s", + "recovery_request": "tcp_compressed_insert_plain_before_schema_recovery_request_rev54460.bytes", + "recovery": "tcp_compressed_insert_plain_before_schema_recovery_rev54460.bytes", + "recovery_packets": [ + { + "code": 1, + "offset": 0, + "name": "", + "rows": 0, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "frames": [], + "native_length": 30, + "native_sha256": "624d5c6e30f14627e2b48f99477009116fbadd01b455aadf502b3040d726775e", + "length": 32 + }, + { + "code": 1, + "offset": 32, + "name": "", + "rows": 1, + "columns": 2, + "schema": [ + [ + "n", + "UInt64" + ], + [ + "s", + "String" + ] + ], + "frames": [], + "native_length": 45, + "native_sha256": "7b111c8bb8757adb1cd64945123d79daa7d80bd63da292080242ddf6f8f73f3d", + "length": 47 + }, + { + "code": 6, + "offset": 79, + "profile": [ + 1, + 1, + 4624 + ], + "applied_limit": 0, + "rows_before_limit": 0, + "obsolete": 1, + "length": 8 + }, + { + "code": 3, + "offset": 87, + "progress": [ + 1, + 1, + 1, + 0, + 0, + 528292 + ], + "length": 9 + }, + { + "code": 1, + "offset": 96, + "name": "", + "rows": 0, + "columns": 0, + "schema": [], + "frames": [], + "native_length": 10, + "native_sha256": "e07d2d762f3e51b34b91f72c628ffbb4f0384ec1c38dd7c35e4b6b4f238f85b4", + "length": 12 + }, + { + "code": 3, + "offset": 108, + "progress": [ + 0, + 0, + 0, + 0, + 0, + 1833 + ], + "length": 8 + }, + { + "code": 5, + "offset": 116, + "length": 1 + } + ], + "verified_http_rows": { + "count": 0, + "length": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + } + } + }, + "files": { + "tcp_compressed_insert_multi_request_rev54460.bytes": { + "length": 334, + "sha256": "12caacf2491f63a0c7cf4c27c82f4234e208a6903558baa8b4cd001ca12e22f6" + }, + "tcp_compressed_insert_multi_schema_rev54460.bytes": { + "length": 121, + "sha256": "2467d56c7f69a3b975af017632800847cc265909a574162879c2302908739aba" + }, + "tcp_compressed_insert_multi_block0_rev54460.bytes": { + "length": 72, + "sha256": "b2207815bc2101aaadf109be482eadee99600766ecff59f0b84bb738bcfc44b7" + }, + "tcp_compressed_insert_multi_block1_rev54460.bytes": { + "length": 4425, + "sha256": "aa6b3e23a874aa1d532d5cc8d4ed07e257ca84e20cd3d81525146406029d950f" + }, + "tcp_compressed_insert_multi_finish_rev54460.bytes": { + "length": 38, + "sha256": "27ba1af6e9b5561c07b5975e29da7019e3f12eec60535f94b4864511b3a4f2ca" + }, + "tcp_compressed_insert_multi_completion_rev54460.bytes": { + "length": 1, + "sha256": "e77b9a9ae9e30b0dbdb6f510a264ef9de781501d7b6b92ae89eb059c5ab743db" + }, + "tcp_compressed_insert_multi_recovery_request_rev54460.bytes": { + "length": 354, + "sha256": "dd7e42707f64404aeae57d8cace558a0e0031d32d4690b7569db949b9d7ae438" + }, + "tcp_compressed_insert_multi_recovery_rev54460.bytes": { + "length": 4566, + "sha256": "f929ef9b43ab77dd9c0073c56b8c237d264808d1660c251d9f6668feb7897be8" + }, + "tcp_compressed_insert_rejected_request_rev54460.bytes": { + "length": 340, + "sha256": "c34448dcb2f067db9d8403765d909e5e9b0f543f5f2d43363657e3948ce3e3a7" + }, + "tcp_compressed_insert_rejected_schema_rev54460.bytes": { + "length": 121, + "sha256": "2467d56c7f69a3b975af017632800847cc265909a574162879c2302908739aba" + }, + "tcp_compressed_insert_rejected_block0_rev54460.bytes": { + "length": 74, + "sha256": "e6e10b9668a06f1be6715a256e185620301db01fc24ff602399d84f9b0580e9d" + }, + "tcp_compressed_insert_rejected_finish_rev54460.bytes": { + "length": 38, + "sha256": "27ba1af6e9b5561c07b5975e29da7019e3f12eec60535f94b4864511b3a4f2ca" + }, + "tcp_compressed_insert_rejected_completion_rev54460.bytes": { + "length": 2462, + "sha256": "9b656b8edfb5a12c2634a597513a959b0a51eec8bd220a037ce6f932e81c9bba" + }, + "tcp_compressed_insert_before_schema_request_rev54460.bytes": { + "length": 305, + "sha256": "cc7594d357fd83942a43e64b32a7dfcb5b129d42a89a40d612fcde5ee6bcf8c1" + }, + "tcp_compressed_insert_before_schema_completion_rev54460.bytes": { + "length": 2326, + "sha256": "d0071e3bed76cbe83278e4fbf07fadf71960bb3b6c0edb3954c710a572f7785c" + }, + "tcp_compressed_insert_before_schema_recovery_request_rev54460.bytes": { + "length": 344, + "sha256": "ee373da34f180cf82bd8e8f3f11756a118c3bc4dbee9fd42bc7a682127732fb9" + }, + "tcp_compressed_insert_before_schema_recovery_rev54460.bytes": { + "length": 195, + "sha256": "e487c1d66f6f4a9faf15a893de842cfc53047ef2793b7abf92801aa0076c22eb" + }, + "tcp_compressed_insert_plain_before_schema_request_rev54460.bytes": { + "length": 291, + "sha256": "5163e00fb9cce4a1785248a5b24612d0b4ae9f81a7468bcdf0815c973e4f94f3" + }, + "tcp_compressed_insert_plain_before_schema_completion_rev54460.bytes": { + "length": 2326, + "sha256": "d0071e3bed76cbe83278e4fbf07fadf71960bb3b6c0edb3954c710a572f7785c" + }, + "tcp_compressed_insert_plain_before_schema_recovery_request_rev54460.bytes": { + "length": 330, + "sha256": "3e682b514d434ccda6493167f01c66586074ebc36990d424660c539c051f4ce6" + }, + "tcp_compressed_insert_plain_before_schema_recovery_rev54460.bytes": { + "length": 117, + "sha256": "0449e46cdc7aa6eb926c42a61856141dbf4bed58f6112f43a169b163f7ba23f9" + } + } +}