Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

* Added the `clickhouse.client.compression` database option to choose the transport compression:
`"lz4"` (the default) or `"none"`. ClickHouse compresses each query response on a single thread,
so `"none"` can be much faster on fast networks.
* Implemented `Statement::execute_schema()` with `DESCRIBE (<query>)`, using the same type mapping
as `Connection::get_table_schema()`. Bound parameters are sent with the `DESCRIBE`.

### Fixed

* Map ClickHouse `Date` to Arrow `Date32` when `output_format_arrow_date_as_uint16` is `0`, the default
since ClickHouse 26.2, so `Connection::get_table_schema()` and `Statement::execute_schema()` match the
data `Statement::execute()` returns. Older servers, which always write `Date` as `UInt16`, are unchanged.
* Retry `Statement::execute()` and `Statement::execute_schema()` for up to about five seconds when
ClickHouse rejects them with `SESSION_IS_LOCKED`, which happened on any query sent right after a
result set on the same connection was dropped before it was fully read.

## [0.1.2] - 2026-11-06

### Fixed
Expand Down
4 changes: 4 additions & 0 deletions agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ A primer for AI agents working on this repository. Read this before doing substa
- **`arrow-ipc::StreamDecoder::finish()` only inspects state, doesn't mutate it.** Safe to call multiple times. It also catches every partial-data / truncated-stream scenario with `IpcError("Unexpected End of Stream")`, so this driver doesn't need to re-validate that.
- **`SchemaRef = Arc<Schema>`.** Existing codebase convention is `Schema::new(...).into()` / `Schema::empty().into()`, *not* `Arc::new(Schema::...)`. The `Arc::new` form is only used here for `Runtime` and `Client`.

- **`Date` in `ArrowStream` depends on the server version.** ClickHouse 26.2 added `output_format_arrow_date_as_uint16` (default `0` → Arrow `date32`); 26.1 and older lack the setting and always write `uint16` (checked against the 25.8, 26.1, 26.2 and 26.10 Docker images). Declared schemas (`get_table_schema`, `execute_schema`) must follow it via `Settings::query()`.

- **A session runs one HTTP request at a time.** A request arriving while another holds the session fails with `SESSION_IS_LOCKED` (373) before any data, e.g. the next query after a result set is dropped unread (the server is still finishing it). `retry_if_session_locked()` in `statement.rs` covers `execute()`/`execute_schema()`; repro in `tests/it/session.rs`.

## Related local repos worth referencing

- `/Users/josemunoz/repos/clickhouse-rs` — the underlying ClickHouse client. Useful files: `src/response.rs` (HTTP error filtering), `src/cursors/*` (fused stream behavior), `src/query.rs` (`fetch_bytes` / `execute`).
Expand Down
14 changes: 14 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ impl Driver for ClickhouseDriver {
password: None,
product_info: self.product_info.clone(),
settings: HashMap::new(),
compression: None,
};

for (key, value) in opts {
Expand Down Expand Up @@ -356,6 +357,8 @@ pub struct ClickhouseDatabase {
password: Option<String>,
product_info: ProductInfo,
settings: HashMap<String, String>,
/// `None` keeps the `clickhouse` crate's default (LZ4).
compression: Option<clickhouse::Compression>,
}

impl Database for ClickhouseDatabase {
Expand Down Expand Up @@ -386,6 +389,10 @@ impl Database for ClickhouseDatabase {
// to a different value at a lower level; `AugmentedClient` covers that.
.with_product_info("adbc_clickhouse", env!("CARGO_PKG_VERSION"));

if let Some(compression) = self.compression {
client = client.with_compression(compression);
}

if let Some(url) = &self.url {
client = client.with_url(url.to_string());
}
Expand Down Expand Up @@ -583,6 +590,9 @@ impl ClickhouseDatabase {
options::PRODUCT_INFO => {
self.product_info = value.try_into()?;
}
options::COMPRESSION => {
self.compression = Some(options::parse_compression(&value.try_string(key)?)?);
}
other => {
return Err(Error::with_message_and_status(
format!("unknown database option {other:?}"),
Expand All @@ -605,6 +615,10 @@ impl ClickhouseDatabase {

match key {
options::PRODUCT_INFO => Ok(self.product_info.to_string()),
options::COMPRESSION => Ok(options::compression_name(
self.compression.unwrap_or(clickhouse::Compression::Lz4),
)
.to_string()),
other => Err(Error::with_message_and_status(
format!("unknown database option {other:?}"),
Status::InvalidArguments,
Expand Down
50 changes: 50 additions & 0 deletions src/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,56 @@ pub const QUERY_ID: &str = "clickhouse.client.query_id";
/// as Arrow binary strings instead.
pub const OUTPUT_STRING_AS_STRING: &str = "clickhouse.client.output_string_as_string";

/// Transport compression between the driver and ClickHouse: `"lz4"` (the default) or `"none"`.
///
/// With `"lz4"`, ClickHouse compresses each query response into LZ4 blocks (the HTTP `compress=1`
/// parameter) and the driver decompresses them; `INSERT` data is compressed by the driver.
/// With `"none"`, data is sent as is.
///
/// ClickHouse compresses a query's response on a single thread, so LZ4 can limit the throughput
/// of one query to well below what a fast network carries. On fast links, `"none"` is often
/// faster; on slow links, `"lz4"` sends fewer bytes.
///
/// May be set on [`ClickhouseDatabase`] only, and applies to every connection it creates.
/// The current value may be read back with [`Optionable::get_option_string()`].
///
/// # Example
/// ```rust
/// use adbc_clickhouse::ClickhouseDriver;
///
/// use adbc_core::{Driver, Optionable};
///
/// let mut driver = ClickhouseDriver::init();
///
/// let db = driver
/// .new_database_with_opts([(adbc_clickhouse::options::COMPRESSION.into(), "none".into())])
/// .unwrap();
///
/// assert_eq!(
/// db.get_option_string(adbc_clickhouse::options::COMPRESSION.into()).unwrap(),
/// "none"
/// );
/// ```
pub const COMPRESSION: &str = "clickhouse.client.compression";

pub(crate) fn parse_compression(value: &str) -> Result<clickhouse::Compression, Error> {
match value.to_ascii_lowercase().as_str() {
"none" => Ok(clickhouse::Compression::None),
"lz4" => Ok(clickhouse::Compression::Lz4),
_ => Err(Error::with_message_and_status(
format!("invalid value for {COMPRESSION:?}: {value:?} (expected \"lz4\" or \"none\")"),
Status::InvalidArguments,
)),
}
}

pub(crate) fn compression_name(compression: clickhouse::Compression) -> &'static str {
match compression {
clickhouse::Compression::None => "none",
_ => "lz4",
}
}

/// Prefix for passing arbitrary [ClickHouse settings] through as driver options.
///
/// An option key of the form `clickhouse.setting.<setting_name>` maps directly to the
Expand Down
70 changes: 69 additions & 1 deletion src/schema.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use adbc_core::error::{Error, Status};
use arrow_schema::{DataType, Field, Fields, Schema, TimeUnit};
use clickhouse::Client;
use clickhouse::query::Query;
use clickhouse_types::DataTypeNode;
use clickhouse_types::data_types::{DateTimePrecision, DecimalType, EnumType};
use serde::{Deserialize, Deserializer};
Expand Down Expand Up @@ -38,6 +39,12 @@ struct Settings {
/// <https://clickhouse.com/docs/operations/settings/formats#output_format_arrow_use_64_bit_indexes_for_dictionary>
#[serde(deserialize_with = "deserialize_bool_setting")]
use_64_bit_indexes_for_dictionary: bool,
/// <https://clickhouse.com/docs/operations/settings/formats#output_format_arrow_date_as_uint16>
///
/// Added in ClickHouse 26.2 with default `0` (`Date` as Arrow `date32`). Older servers lack the
/// setting and always write `Date` as `UInt16`, so a missing setting means `true`.
#[serde(deserialize_with = "deserialize_bool_setting_default_true")]
date_as_uint16: bool,
}

impl Default for Settings {
Expand All @@ -56,6 +63,7 @@ impl Default for Settings {
low_cardinality_as_dictionary: false,
use_signed_indexes_for_dictionary: false,
use_64_bit_indexes_for_dictionary: false,
date_as_uint16: true,
}
}
}
Expand All @@ -70,6 +78,7 @@ impl Settings {
"low_cardinality_as_dictionary",
"use_signed_indexes_for_dictionary",
"use_64_bit_indexes_for_dictionary",
"date_as_uint16",
];

let mut sql = "SELECT ".to_string();
Expand Down Expand Up @@ -166,6 +175,49 @@ pub async fn of_table(
Ok(Schema::new(fields))
}

/// Describe the result columns of a query prepared as `DESCRIBE (<query>)`.
///
/// The columns are mapped with the same type mapping (and the same server settings) as
/// [`of_table()`], so the schema matches the Arrow stream ClickHouse returns for the query.
pub async fn of_query(client: &Client, query: Query) -> adbc_core::error::Result<Schema> {
let mut cursor = query.fetch::<Column>().map_err(|e| {
Error::with_message_and_status(
format!("error beginning fetch of query schema: {e}"),
Status::Internal,
)
})?;

let mut columns = Vec::new();
while let Some(column) = cursor.next().await.map_err(|e| {
Error::with_message_and_status(
format!("error retrieving query schema: {e}"),
Status::Internal,
)
})? {
columns.push(column);
}
drop(cursor);

// Both requests carry the connection's session ID, and ClickHouse rejects a request that
// overlaps another on the same session, so finish `DESCRIBE` before querying the settings.
let settings = Settings::query(client).await?;

let mut fields = Vec::with_capacity(columns.len());
for column in &columns {
let field = column_to_arrow(column, &settings).map_err(|e| Error {
message: format!(
"error mapping query column {:?}: {}",
column.name, e.message
),
..e
})?;

fields.push(field);
}

Ok(Schema::new(fields))
}

fn column_to_arrow(column: &Column, settings: &Settings) -> Result<Field> {
let ch_type = DataTypeNode::new(&column.r#type).map_err(|e| {
Error::with_message_and_status(
Expand Down Expand Up @@ -240,7 +292,8 @@ fn ch_type_to_arrow(
// a canonical extension type in Arrow: https://arrow.apache.org/docs/format/CanonicalExtensions.html#uuid
// DataTypeNode::UUID => {}
// https://github.com/ClickHouse/ClickHouse/blob/3196ab525aa6f1fef8d367db7610b765f7737f01/src/Processors/Formats/Impl/CHColumnToArrowColumn.cpp#L86
DataTypeNode::Date => Ok(DataType::UInt16),
DataTypeNode::Date if settings.date_as_uint16 => Ok(DataType::UInt16),
DataTypeNode::Date => Ok(DataType::Date32),
DataTypeNode::Date32 => Ok(DataType::Date32),
// https://github.com/ClickHouse/ClickHouse/blob/3196ab525aa6f1fef8d367db7610b765f7737f01/src/Processors/Formats/Impl/CHColumnToArrowColumn.cpp#L87C1-L87C114
DataTypeNode::DateTime(_) => Ok(DataType::UInt32),
Expand Down Expand Up @@ -447,3 +500,18 @@ where
_ => Err(Error::custom(format!("expected \"0\" or \"1\", got {s:?}"))),
}
}

/// Like [`deserialize_bool_setting`], but a missing setting means `true`.
fn deserialize_bool_setting_default_true<'de, D>(deserializer: D) -> Result<bool, D::Error>
where
D: Deserializer<'de>,
{
use serde::de::Error;

match Option::<String>::deserialize(deserializer)?.as_deref() {
None => Ok(true),
Some("0") => Ok(false),
Some("1") => Ok(true),
Some(s) => Err(Error::custom(format!("expected \"0\" or \"1\", got {s:?}"))),
}
}
100 changes: 84 additions & 16 deletions src/statement.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use clickhouse::query::Query;

use crate::options::OptionValueExt;
use crate::writer::ArrowStreamWriter;
use crate::{AugmentedClient, Result, TokioContext, options, random_id};
use crate::{AugmentedClient, Result, TokioContext, options, random_id, schema};
use clickhouse::_priv::sql_escape_identifier;
use clickhouse_ext_arrow::ArrowQueryExt;

Expand Down Expand Up @@ -251,13 +251,50 @@ impl Statement for ClickhouseStatement {
execute_streaming_ingest(&self.client, &self.tokio, ingest, stream)
}

/// # Not Implemented
/// Currently will return an error with [`Status::NotImplemented`].
/// Get the schema of the query's result set without executing the query.
///
/// See the tracking issue for details and subscribe for updates:
/// <https://github.com/ClickHouse/adbc_clickhouse/issues/16>
/// This runs `DESCRIBE (<query>)` on the server and maps the result columns with the same
/// type mapping as [`Connection::get_table_schema()`][adbc_core::Connection::get_table_schema],
/// so the schema matches the one [`execute()`][Self::execute] returns. Bound parameters
/// (from [`bind()`][Self::bind]) are sent with the `DESCRIBE`.
///
/// Returns [`Status::InvalidState`] if no query is set, or if the query is an `INSERT`
/// that would stream bound data.
fn execute_schema(&mut self) -> adbc_core::error::Result<Schema> {
err_unimplemented!("ClickhouseStatement::execute_schema()")
let StatementMode::SqlQuery(sql) = &self.mode else {
return Err(Error::with_message_and_status(
"execute_schema requires a SQL query",
Status::InvalidState,
));
};
if is_streaming_insert(sql) {
return Err(Error::with_message_and_status(
"execute_schema requires a query that returns a result set",
Status::InvalidState,
));
}

let _guard = self.tokio.enter();

retry_if_session_locked(|| {
// This statement's query ID is reused for its eventual data query. Give each metadata
// request its own ID, so ClickHouse does not reject the data query (or the second
// metadata request) while an earlier request with the same ID is still being cleaned up.
let mut schema_client = Client::clone(&self.client);
schema_client.set_setting(options::as_setting::QUERY_ID, random_id("schema"));
let mut settings_client = schema_client.clone();
settings_client
.set_setting(options::as_setting::QUERY_ID, random_id("schema_settings"));

// Raw so a literal `?` is not treated as a bind placeholder.
let mut query = schema_client.query_raw(&format!("DESCRIBE ({sql})"));
if let Some(BindType::Single(batch)) = &self.bind {
query = bind_record_batch(query, batch.clone())?;
}

self.tokio
.block_on(schema::of_query(&settings_client, query))
})
}

/// # Not Implemented
Expand Down Expand Up @@ -955,18 +992,49 @@ fn fetch_blocking(
// but that results in a larger generated future type
let _guard = tokio.enter();

// Raw so a literal `?` is not treated as a bind placeholder;
// parameters are bound server-side via `Query::param()`.
let mut query = client.query_raw(sql);
query = bind_query(query, bind)?;
let fetch = |bind: Option<BindType>| -> Result<ArrowStreamReader, Error> {
// Raw so a literal `?` is not treated as a bind placeholder;
// parameters are bound server-side via `Query::param()`.
let mut query = client.query_raw(sql);
query = bind_query(query, bind)?;

let cursor = query.fetch_arrow().map_err(|e| {
Error::with_message_and_status(format!("error executing query: {e}"), Status::Internal)
})?;
let cursor = query.fetch_arrow().map_err(|e| {
Error::with_message_and_status(format!("error executing query: {e}"), Status::Internal)
})?;

tokio
.block_on(ArrowStreamReader::begin(tokio.clone(), cursor))
.map_err(Into::into)
tokio
.block_on(ArrowStreamReader::begin(tokio.clone(), cursor))
.map_err(Into::into)
};

match bind {
// A streamed bind cannot be replayed.
Some(BindType::Stream(stream)) => fetch(Some(BindType::Stream(stream))),
Some(BindType::Single(batch)) => {
retry_if_session_locked(|| fetch(Some(BindType::Single(batch.clone()))))
}
None => retry_if_session_locked(|| fetch(None)),
}
}

/// ClickHouse runs one request at a time per session and rejects a request that arrives while
/// another one holds the session (`SESSION_IS_LOCKED`, code 373). That happens, for example, when
/// a result set is dropped before it is fully read: the server is still finishing that query when
/// the next request arrives. The error comes before any data, so retry for about five seconds.
fn retry_if_session_locked<T>(mut request: impl FnMut() -> Result<T, Error>) -> Result<T, Error> {
const RETRY_LIMIT: u32 = 30;
let mut attempt = 0;
loop {
match request() {
Err(e) if attempt < RETRY_LIMIT && e.message.contains("SESSION_IS_LOCKED") => {
// 5, 10, 20, ... ms, capped at 200 ms.
let delay_ms = (5u64 << attempt.min(6)).min(200);
std::thread::sleep(std::time::Duration::from_millis(delay_ms));
attempt += 1;
}
result => return result,
}
}
}

fn execute_blocking(
Expand Down
Loading