feat(l2): @clickhouse/client byte-compat embedded façade (Layer 2) by ShawnChen-Sirius · Pull Request #49 · chdb-io/chdb-node

ShawnChen-Sirius · 2026-06-17T23:02:02Z

What

Adds Layer 2: an embedded-only façade that mirrors the @clickhouse/client surface (createClient, query/command/exec/insert/ping/stream, ResultSet, the typed error hierarchy) on top of the v3 Layer 1 connection. The goal is byte-compatible behavior so code written against clickhouse-js runs unchanged against an in-process chDB.

Shape

src/layer2/*.ts (compiled to dist/layer2) — a thin forwarder to Layer 1.
chdb://memory clients share one process-wide reference-counted Session; chdb:///path clients share one per absolute path. Opening a different on-disk path while one is live surfaces ChdbConnectionError (single active connection per process).
query() is eager-buffered via Layer 1 queryAsync so errors surface at await query() (byte-compat); stream() replays the buffer through a newline transform.
Int64/UInt64 → string in JSON-family output (injects output_format_json_quote_64bit_integers=1) to match server/clickhouse-js and avoid silent JSON.parse precision loss.
Typed error hierarchy re-exported from the package root so instanceof holds against the same module instance the runtime throws from.

Tests

New suites under test/v3/layer2/* (formats / url / sql_guard / result_set / client / config / errors / adversarial / compat-types).
Backend-swappable upstream/conformance.test.ts (embedded by default, or CHDB_UPSTREAM_BACKEND=server) asserting true ClickHouse semantics.
Opt-in parity.test.ts (CHDB_PARITY_URL) + a non-gating clickhouse-js literal-suite import-rewrite harness, with a layer2.yml CI workflow.

Full v3 suite: 257 passing, 16 skipped (opt-in server parity), 0 failing. Typecheck clean.

Rebase note

Rebased onto current main (now carries the global session-cleanup safety net). That safety net exposed two harness issues, fixed in the last commit:

vitest evaluated the CJS entrypoint twice (transformed import vs the require('../../index.js') the compiled Layer 2 uses to reach Layer 1), so the cleanup hook drained a different instance than Layer 2's sessions — a timed-out heavy query then leaked its connection and cascaded "only one active data directory" failures. Fixed by externalizing the entrypoint so every importer shares one Node-cached instance.
The conformance test held a beforeAll-shared client that the now-effective cleanup tears down after the first case; switched to a per-test client (behavior-preserving — each case is self-contained).

Honest gaps (follow-ups, not this PR)

True zero-copy streaming (currently buffers the whole result).
The clickhouse-js literal-jest harness is non-gating / server-coupled; the verified gate is upstream/conformance.test.ts.
Bun/Deno runtime matrix runs in CI only.
Parity server pinned to CH 24.8 — bump to track chdb-core's underlying CH version.

🤖 Generated with Claude Code

Layer 2 mirrors the @clickhouse/client surface (createClient / query / command / exec / insert / ping / stream, ResultSet, the typed error hierarchy) over an in-process chDB connection, so code written against clickhouse-js runs unchanged in embedded mode. chdb://memory clients share one reference-counted Session; query() is eager-buffered so errors surface at await; JSON-family output injects output_format_json_quote_64bit_integers=1 to keep 64-bit ints lossless, matching clickhouse-js's own default. Inserts and query parameters are encoded exactly as clickhouse-js does, decoded by the engine rather than re-implemented: - insert serializes row arrays to a FORMAT-tailed dataset (JSONEachRow for object rows, JSONCompactEachRow for positional arrays, inferred when omitted) instead of a SQL VALUES literal, so arrays / maps / tuples / Nested / JSON round-trip; an explicit null is ClickHouse NULL, only an undefined cell is rejected. - a clickhouse-js-faithful query-parameter formatter (src/layer2/params.ts): top-level strings unquoted and null as the TSV token, but inside Array/Tuple/Map strings are quoted, null is the NULL keyword and booleans are TRUE/FALSE; Date is a Unix timestamp; tuples use a TupleParam wrapper (exported here, and a clickhouse-js TupleParam is accepted structurally for migration). The formatted map is bound verbatim via a preformatted option on Layer 1's queryBindAsync, leaving Layer 1's own serializer untouched. Parameter handling is type-aware via the engine, not guarded client-side: an out-of-range Int64 is a typed error while the same value binds fine to a Float64. Includes the Layer 2 unit/behavior tests and a vitest config fix that externalizes the CJS entrypoint so the suite's session-cleanup safety net and the tests share a single module instance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s's own suite Two harnesses turn the byte-compat claim into something checked, not asserted. Parity (test/v3/layer2/upstream/conformance.test.ts + parity.test.ts) runs the same queries through embedded chDB and a real clickhouse-server and compares the decoded output. The server image tracks chDB's kernel (ClickHouse 26.5); both clients set output_format_json_quote_64bit_integers=1 (clickhouse-js's own suite default) so 64-bit ints are lossless on both sides. Clients are created per test so the global session-cleanup afterEach cannot tear down a shared connection. Upstream-literal (scripts/upstream-suite) runs clickhouse-js's OWN integration suite against embedded chDB: it clones clickhouse-js at the installed client's version, redirects the suite's client factory to chdb://memory, and runs the specs on vitest serially. The previous harness ran jest, but clickhouse-js migrated to vitest — so it had been parsing zero specs and silently passing. skip-list.json drops whole files for capabilities embedded lacks (HTTP transport, sockets, compression, RBAC, server runtime, cluster, TLS); expectations.patch marks the per-case embedded-vs-server divergences it.fails (documented in the suite README) so they must keep failing without hiding a regression. Both jobs gate CI (continue-on-error removed). Against clickhouse-js 1.20.0 on embedded chDB: 175 passing, 38 expected-fail, 12 skipped, 0 unexpected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Two independent failures on the PR layer2 workflow: 1. parity (22/23 red) — the clickhouse/clickhouse-server:26.5 image's entrypoint writes a users.d snippet requiring a password for `default` when CLICKHOUSE_PASSWORD is unset, so every parity query returned code 194 REQUIRED_PASSWORD while embedded chDB returned the correct semantic code (e.g. 60 UNKNOWN_TABLE). Locally the worklog's `clickhousectl local install 26.5` doesn't add that snippet, hence the local 23/23 green. Set CLICKHOUSE_SKIP_USER_SETUP=1 on the service so the image keeps the upstream password-less default — the service is sealed inside the GH Actions network, no real auth needed. 2. upstream-literal (6 "Expect test to fail") — 4 DateTime/Date cases and 2 others (floating point types, JS Date objects) were marked `it.fails` in expectations.patch but actually pass in CI: GitHub runners run UTC, which eliminates the DateTime session-timezone divergence the worklog deferred as Stage B. The gate is working as designed (an unexpected pass is a hard failure), and these cases have simply graduated from "known divergence" to "byte-compat under UTC". Drop the 6 hunks from expectations.patch and pin TZ=UTC on the upstream step so the contract reproduces off CI. Remaining 32 markers stay; categories 4 and 6 in the README narrowed accordingly. The macos-15-intel/20.x check is unrelated (a native chDB abort-recovery crash on that one matrix cell after the abort-storm stress test); a re-run should clear it and the underlying robustness issue belongs to chdb-core, not this PR. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>

ShawnChen-Sirius · 2026-06-18T02:36:27Z

@chibugai, review this PR.

Copilot

Pull request overview

Adds Layer 2: an embedded-only, @clickhouse/client-shaped façade implemented on top of the existing v3 (Layer 1) connection/session model, plus a substantial conformance/parity test harness and CI wiring to validate byte-compat behavior.

Changes:

Introduces src/layer2/* (client, ResultSet, URL parsing, SQL guard, settings prefixing, error mapping, and type surface) and re-exports Layer 2 + typed errors from the package entrypoints.
Updates Layer 1 async bind path to support Layer 2 “preformatted” query params without re-serializing.
Adds Layer 2 test suites + upstream conformance/parity harness, plus a dedicated CI workflow and npm scripts to run them.

Reviewed changes

Copilot reviewed 39 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vitest.config.ts	Adjusts vitest dependency handling to avoid double-evaluating the CJS entrypoint.
test/v3/querybind.test.ts	Updates bind behavior expectations (undefined rejected; null binds as NULL).
test/v3/layer2/url.test.ts	Adds URL parsing tests for `chdb://` handling.
test/v3/layer2/upstream/README.md	Documents upstream conformance harness and skip list rationale.
test/v3/layer2/upstream/conformance.test.ts	Adds backend-swappable conformance tests for true ClickHouse semantics.
test/v3/layer2/upstream/_backend.ts	Implements backend switch between embedded and real server client.
test/v3/layer2/sql_guard.test.ts	Adds tests for cluster-topology SQL guard behavior.
test/v3/layer2/result_set.test.ts	Adds ResultSet/Row behavior tests including chunk boundary handling.
test/v3/layer2/parity.test.ts	Adds opt-in parity checks against a real ClickHouse server.
test/v3/layer2/formats.test.ts	Adds drift-guard tests for format classification lists.
test/v3/layer2/errors.test.ts	Adds error mapping/instanceof boundary tests.
test/v3/layer2/config.test.ts	Adds config arbitration tests (ignored fields, timeouts, settings).
test/v3/layer2/compat-types.test.ts	Adds TS-level structural compatibility checks against `@clickhouse/client`.
test/v3/layer2/client.test.ts	Adds end-to-end client method behavior tests.
test/v3/layer2/adversarial.test.ts	Adds adversarial tests for URLs, params, inserts, and lifecycle invariants.
src/serialize.ts	Updates Layer 1 param formatting to allow `null` (TSV `\\N`) and reject `undefined`.
src/layer2/url.ts	Implements manual parsing for `chdb://` URLs (+ `?database=`).
src/layer2/types.ts	Defines Layer 2 structural types mirroring `@clickhouse/client`.
src/layer2/sql_guard.ts	Adds lexical SQL guard for embedded-unsupported cluster topology features.
src/layer2/settings.ts	Implements engine `SET ...;` prefix building for `clickhouse_settings`.
src/layer2/result_set.ts	Implements buffer-backed ResultSet with clickhouse-js-like semantics.
src/layer2/params.ts	Implements clickhouse-js-style query param serialization.
src/layer2/layer1.ts	Adds lazy typed accessor for the Layer 1 CJS runtime entrypoint.
src/layer2/index.ts	Exposes Layer 2 surface for re-export from the package root.
src/layer2/formats.ts	Adds upstream-compatible format classification helpers.
src/layer2/errors.ts	Adds Layer 2 error model: ClickHouseError + embedded boundary errors.
src/layer2/error_map.ts	Implements single-point engine error -> ClickHouseError mapping.
src/layer2/create_client.ts	Adds Layer 2 createClient + embedded connection registry/refcounting.
src/layer2/client.ts	Implements the Layer 2 client façade over Layer 1 sessions.
scripts/upstream-suite/skip-list.json	Defines upstream integration-suite file-level skips for embedded mode.
scripts/upstream-suite/rewrite-and-run.mjs	Implements upstream suite clone+rewrite+run harness.
scripts/upstream-suite/README.md	Documents how to run/maintain the upstream literal-suite harness.
scripts/upstream-suite/expectations.patch	Marks known embedded-vs-server divergences as expected failures.
README.md	Documents Layer 2 drop-in usage and links to the detailed compatibility doc.
package.json	Adds scripts for parity/upstream runs and adds optional peer dep on `@clickhouse/client`.
package-lock.json	Updates lockfile for new dev deps + version bump.
index.mjs	Re-exports Layer 2 + typed error hierarchy from the ESM entrypoint.
index.js	Re-exports Layer 2 + typed errors from the CJS entrypoint; adds preformatted bind support.
index.d.ts	Re-exports Layer 2 declaration surface and typed errors.
docs/layer2-clickhouse-js-compat.md	Adds detailed Layer 2 migration guide, capability matrix, and semantics reference.
.github/workflows/layer2.yml	Adds dedicated CI workflow for parity + upstream literal suite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@throws

Builds on the CI fix (UTC + skip-user-setup); does not touch the workflow. - A failed `USE <database>` no longer poisons the client: the memoized USE promise is cleared on rejection so the next operation retries instead of replaying the cached error. - insert() now applies clickhouse_settings (client + per-call) and the request_timeout / abort_signal options, matching query/command/exec — settings were previously dropped on the INSERT path. With settings now applied, the CustomSeparated inserts, nested-json input, insert_deduplication_token and the unknown-settings error case pass against clickhouse-js's suite, so their it.fails marks are removed from expectations.patch (now 25 expected-fail). - The upstream skip-list now matches a spec's basename rather than its full path, per the documented "basename substring" intent. - Doc/comment cleanups: formatParamValue's @throws no longer claims null throws (null binds as the TSV NULL token; only undefined throws); removed the unused rootIndex/fileURLToPath in vitest.config; corrected the sql_guard comment that claimed a negative lookahead it does not use. Not changed: Layer 2's parameter formatter maps `undefined` to the TSV null token rather than rejecting it — this mirrors clickhouse-js's formatQueryParams exactly, which is the point of the byte-compat layer; Layer 1's binder still rejects it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 39 out of 41 changed files in this pull request and generated 2 comments.

@throws

Builds on the CI fix (UTC + skip-user-setup); does not touch the workflow. - A failed `USE <database>` no longer poisons the client: the memoized USE promise is cleared on rejection so the next operation retries instead of replaying the cached error. - insert() now applies clickhouse_settings (client + per-call) and the request_timeout / abort_signal options, matching query/command/exec — settings were previously dropped on the INSERT path. With settings now applied, the CustomSeparated inserts, nested-json input, insert_deduplication_token and the unknown-settings error case pass against clickhouse-js's suite, so their it.fails marks are removed from expectations.patch (now 25 expected-fail). - The upstream skip-list now matches a spec's basename rather than its full path, per the documented "basename substring" intent. - makeRowTransform now flushes a final non-newline-terminated fragment in a flush handler, so a last row without a trailing newline is no longer silently dropped (chDB row formats always trail a newline, but the transform was copied from clickhouse-js, which flushes). Added a regression test. - Doc/comment cleanups: formatParamValue's @throws no longer claims null throws (null binds as the TSV NULL token; only undefined throws); removed the unused rootIndex/fileURLToPath in vitest.config; corrected the sql_guard comment that claimed a negative lookahead it does not use. Not changed: Layer 2's parameter formatter maps `undefined` to the TSV null token rather than rejecting it — this mirrors clickhouse-js's formatQueryParams exactly, which is the point of the byte-compat layer; Layer 1's binder still rejects it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A docs/design companion to architecture.md (positioning) and the user-facing compat guide: what Layer 2 is responsible for, the design principles, the module map, a commit-ordered implementation map, the key implementation decisions (connection/session model, FORMAT-tailed insert, the clickhouse-js-faithful parameter formatter + TupleParam, settings/error handling, the vitest single-instance gotcha), the verification strategy (conformance / parity / clickhouse-js's own suite), the honest embedded-vs-server differences, and the Stage B follow-ups. Mirrors the existing layer1-native-binding.md guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The heavy storm/race cases run dozens of multi-second queries in sequence and take ~15-30s on a fast runner. On a slow/loaded macOS runner the "open a new session while a default-conn query is in flight (60x)" case exceeded vitest's 30s default and was killed mid-flight. Its in-flight query uses plain queryAsync (no signal/timeout), so it is not tracked in pendingNativeOps and the global afterEach drain cannot absorb it; the abandoned op then collides with the next test on the single in-process engine and aborts it (code 236), cascading failures into every later file. It flaked twice, on two different runners (macos-15-intel/22.x, macos-14/18.x), while the layer2 parity/upstream jobs stayed green. Give the four heavy storm/race tests a generous explicit timeout (STRESS_TIMEOUT_MS = 120_000) so a slow runner finishes them instead of killing them mid-flight. No change to what they exercise. Follow-up (Layer 1): track plain async ops too, so an abandoned op is always drained and a timeout can never poison the engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the single-active-connection registry with one that allows N independent connections to the same process-wide EmbeddedServer path, so same-path Sessions (and Layer 2 clients) execute queries in parallel instead of serializing on one shared ChdbClient's per-instance mutex. - native (lib/chdb_node.cpp): g_active single slot -> registry of N connections per bound path; a different data directory is still rejected; the last connection out unbinds the server. Guard mutex added. - index.js: each Session owns its own native connection (also fixes a latent same-path clobber). Non-parameterized async queries run in parallel across connections. Parameterized queries are serialized PROCESS-WIDE (one globalParamChain), because the bundled libchdb does not isolate parameter set/reset per connection under true parallelism — concurrent param queries on different connections otherwise clobber (456 Substitution not set) and can fatally abort the engine (236). The GIL hides this in thread-based bindings (chdb-python); libuv exposes it. - layer2 (create_client.ts): each client gets its own connection over a shared path (one refcounted temp dir for chdb://memory), preserving cross-client data visibility while gaining parallelism. - test/v3/multi-connection.test.ts: same-path coexistence, data sharing, different-path rejection, param correctness under concurrency, and real parallelism (~3.7x on 4 cores). - docs: layer1-native-binding.md §9.6.1 documents the model and root cause. This matches chdb-python's model (one ChdbClient per Connection). The prior binding refcount-collapsed every same-path Session onto one ChdbClient, whose client_mutex serialized all queries — which is why node did not parallelize while chdb-python did (verified: the same libchdb runs concurrent chdb_query_n in parallel from independent connections). Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>

formatParamValue rejected a top-level null/undefined with ChdbBindError. The reference client (@clickhouse/client-common formatQueryParams) instead emits the TSV null marker \N at the top level (and the NULL keyword when nested), which the server binds as SQL NULL for Nullable columns. chdb-node is a byte-compat @clickhouse/client facade (chdb-io#49, chdb-io#52), so throwing was a divergence, not an engine limitation: the native chdb_query_with_params path already accepts \N as NULL (verified for Nullable(String)/Int64). Emit \N for a top-level null/undefined; nested nulls already render as the NULL keyword via serializeValue. A null bound to a non-Nullable placeholder is now rejected by the engine (a CHDB_* query error) rather than a client-side ChdbBindError, matching the HTTP client + server. Removes the two NULL-binding entries from the clickhouse-js parity skip list; both now pass against @clickhouse/client 1.23.0. Fixes chdb-io#67 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ShawnChen-Sirius mentioned this pull request Jun 18, 2026

Proposal: chdb-node 3.x layered API plan: native reliability, ClickHouse client compatibility, and TypeScript fluent query layer #39

Closed

ShawnChen-Sirius and others added 2 commits June 18, 2026 14:13

ShawnChen-Sirius force-pushed the feat/layer2-clickhouse-js-compat branch from 664de56 to d0487c1 Compare June 18, 2026 02:16

ShawnChen-Sirius force-pushed the feat/layer2-clickhouse-js-compat branch from f857f98 to fe175e7 Compare June 18, 2026 02:35

chibugai reviewed Jun 18, 2026

View reviewed changes

Comment thread scripts/upstream-suite/rewrite-and-run.mjs Resolved

Comment thread src/layer2/client.ts Resolved

Comment thread src/serialize.ts Resolved

ShawnChen-Sirius requested a review from Copilot June 21, 2026 22:47

Copilot started reviewing on behalf of ShawnChen-Sirius June 21, 2026 22:47

View session

Copilot AI reviewed Jun 21, 2026

View reviewed changes

Comment thread vitest.config.ts Resolved

Comment thread src/layer2/client.ts Resolved

Comment thread src/layer2/params.ts Resolved

Comment thread src/layer2/sql_guard.ts Resolved

ShawnChen-Sirius requested review from chibugai and Copilot June 22, 2026 00:34

Copilot started reviewing on behalf of ShawnChen-Sirius June 22, 2026 00:34

View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread src/layer2/result_set.ts Resolved

Comment thread test/v3/layer2/result_set.test.ts Resolved

ShawnChen-Sirius and others added 2 commits June 22, 2026 13:17

ShawnChen-Sirius force-pushed the feat/layer2-clickhouse-js-compat branch from cdf060a to 19776ef Compare June 22, 2026 01:19

ShawnChen-Sirius and others added 2 commits June 22, 2026 13:42

ShawnChen-Sirius force-pushed the feat/layer2-clickhouse-js-compat branch from 73036be to b01ae32 Compare June 22, 2026 06:07

ShawnChen-Sirius mentioned this pull request Jun 23, 2026

feat(l1): multi-connection model + async-stress timeout stabilization #51

Merged

ShawnChen-Sirius closed this Jun 23, 2026

This was referenced Jul 2, 2026

queryBind rejects null/undefined params instead of binding SQL NULL (diverges from @clickhouse/client) #67

Closed

Bind null query params as SQL NULL (\N), matching @clickhouse/client #68

Merged

Navigation Menu

Repository navigation

feat(l2): @clickhouse/client byte-compat embedded façade (Layer 2) - #49

feat(l2): @clickhouse/client byte-compat embedded façade (Layer 2)#49
ShawnChen-Sirius wants to merge 7 commits into
mainfrom
feat/layer2-clickhouse-js-compat

ShawnChen-Sirius commented Jun 17, 2026

Uh oh!

ShawnChen-Sirius commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ShawnChen-Sirius commented Jun 17, 2026

What

Shape

Tests

Rebase note

Honest gaps (follow-ups, not this PR)

Uh oh!

ShawnChen-Sirius commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants