Skip to content

Pin chdb-core v26.11.0-rc.2 - #99

Merged
wudidapaopao merged 7 commits into
chdb-io:mainfrom
wudidapaopao:bump-engine-v26.11.0-rc.1
Nov 5, 2026
Merged

wudidapaopao merged 7 commits into
chdb-io:mainfrom
wudidapaopao:bump-engine-v26.11.0-rc.1

Conversation

@wudidapaopao

@wudidapaopao wudidapaopao commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Moves the engine pin to chdb-core v26.11.0-rc.2 so the release candidate gets validated here. rc.2 is rc.1 plus chdb-core#312, on the same ClickHouse baseline (master def2fdf4caa, 26.11.1.1). It is a release candidate: it is not on PyPI, and the full release will be v26.11.1. chdb-core's check_bindings.yml only dispatches the engine release check for full releases, so this PR does that check's edit by hand. The PR first pinned rc.1; the third commit moves it to rc.2. The fifth to seventh fix an exit crash on the standalone connection (below).

What changed

  • update_libchdb.sh: CHDB_ENGINE_PIN=v26.11.0-rc.2. LIBCHDB_NPM_VERSION and the four @chdb/lib-* optionalDependencies in package.json become 26.11.0-rc.2.1. That is the scheme's value for a first packaging of an rc engine, and scripts/check-version-scheme.mjs passes. These are the lines engine-release-check.yml's propose_bump rewrites. The publish workflow already sends an -rc. subpackage to the rc dist-tag, so nothing else changes for an rc.
  • Prose that names the pin, the same three places Move the engine pin to v26.7.3 #94 updated: README's durable section, the header of chdb-node-test.yml, and the still-to-do item in docs/design/durable-control-plane.md. The update_libchdb.sh comment gives this build's chdb_version() string (26.11.0-rc.2) and the min_reader floor that replaced exact matching: an object written on rc.2 is refused by rc.1 and earlier.
  • tests/clickhouse-js/skip_list.json: four parity cases skipped, each with its reason (details below).
  • test/v3/engine-exit.test.ts: a new case. A process that used a MergeTree table and exits with its session still open must leave stderr empty. It fails on rc.1 (see below) and passes on rc.2.
  • Exit fix (index.js, lib/chdb_node.cpp, two more engine-exit.test.ts cases): the process 'exit' hook now also closes the standalone default connection and stops the engine before C exit(). Details below.

The pin needs no addon change: rc.2's chdb.h differs from rc.1's only in CHDB_VERSION. Between v26.7.3 and rc.1, chdb.h gained chdb_insert_arrow_array / chdb_insert_arrow_stream, which the addon doesn't use, plus doc comments.

Parity skips (second commit)

The connection / parity job runs @clickhouse/client 1.23.0's suite. Four of its cases fail on every engine from v26.9.0 on. I ran the runner against v26.7.3 (all pass), v26.9.0, rc.1 and rc.2 (the same 4 fail on each):

  • error_parsing "unknown identifier" / "unknown table": the analyzer now writes . In scope with a capital I, and the 1.23.0 regex is case-sensitive. Code and type still parse correctly.
  • date_time "DateTime64(3) … (different timezone)" and data_types "custom JSON handling (BigInt and Date)": a JSON number of epoch milliseconds inserted into DateTime64(3) now saturates to 9999-12-31 23:59:59.000. I confirmed this with plain SQL (format(JSONEachRow, 'd DateTime64(3)', '{"d":1662328969123}')). The same value sent as a JSON string still parses. chdb-node's own serializers send Date as a string, so this only affects callers who put raw epoch-ms numbers into JSONEachRow themselves.

Upstream fixed both in ClickHouse/clickhouse-js#1025 (released in client-1.24.0): it relaxed the regex and skips the DateTime64 cases on servers that clamp. I tried moving the runner to client-1.24.1. That fixes these four but brings three other failures (should not destroy the socket under the next request after the idle TTL, and two Array(Date) query-binding cases), and those also fail on v26.7.3, so they are not caused by this engine. That sync belongs in its own PR. Each skip entry says to drop it when syncedAgainst moves.

Exit fix (fifth to seventh commits)

process.exit() with MergeTree merges in flight on the standalone query() connection crashed in exit(): SIGSEGV, or mutex lock failed: Invalid argument, after chDB's crash handler waited about 30 s. On rc.2 that happened 6 times out of 6. It is not new in rc.2; rc.1 hung instead. The JS 'exit' hook closed only Sessions, and Node runs no environment cleanup hooks on process.exit(). So the default connection was first closed by the addon's std::atexit backstop. That handler is registered at the first connect, and atexit handlers run in reverse order of registration, so by then every static constructed later, including the ones queries and merges construct, had been destroyed. gdb shows the crash in chdb_close_conn called from shutdown_on_process_exit.

  • The 'exit' hook is installed when index.js loads, not with the first Session. After closing the sessions it calls a new native ExitTeardown. That closes every connection no async call is running on (the addon's existing in-flight count) and then calls chdb_shutdown(), all before exit() while every static is alive. Main thread only: the connections and the engine are process-wide, so a worker that exits must not close them under the main thread. Deno.exit() skips the 'exit' event and dispatches unload, so under Deno the hook listens for that too.
  • A connection with a call still running stays open. Closing it under the call aborts the engine, and Node joins that libuv thread only after the 'exit' event. For that case the backstop is registered again, so exit() runs it first: after the join, before the destroyed statics.
  • chdb_shutdown() is called at most once between the exit paths (sixth commit). It keeps a function-local static mutex that its first call constructs. Called first from the 'exit' event, that mutex is destroyed before the atexit backstop runs, and a second call from the backstop locked the destroyed mutex. glibc lets that through. macOS aborts with mutex lock failed: Invalid argument, which failed every macOS job on the fifth commit. gdb now shows one call at exit, where there were two.
  • While a worker_threads Worker has the addon loaded, the 'exit' teardown closes nothing (seventh commit). A worker may be using any connection, and synchronous calls are not in the in-flight count. Node stops workers only after the 'exit' event, so the fifth commit's teardown freed a worker's Session under its next s.query(), a SIGSEGV found in review. The addon now counts the environments that loaded it. While more than one is alive, it only registers the backstop again, the same as for a busy connection. By the time exit() runs it, Node has stopped and joined the workers. A new engine-exit.test.ts case runs synchronous Session queries in a worker while the main thread calls process.exit(0). It fails on the sixth commit and passes on the seventh.
  • The atexit backstop stays as the fallback. Its comment said it ran before the engine's own exit handlers. Since rc.2 the engine registers its teardown on every start, so after a restart that teardown runs first, and the backstop also runs after every static constructed after the first connect. The comment now says so.
  • Behaviour change for 'exit' listeners. The hook is installed when require('chdb') runs. Any process.on('exit') listener added after that runs after the hook, and now finds the engine stopped: a standalone query() there throws Failed to acquire default connection, and a new Session fails too. Such listeners already found their sessions closed. Listeners added before require('chdb') run first and are not affected.
  • Two engine-exit.test.ts cases in the fifth commit: process.exit() with merges in flight on the default connection, and with a 10M-row insert still running on it. Both check that merges were really running (and the insert really unfinished) when the exit began. Both fail without the fix. The first crashes; the second crashed or hung until killed (5 of 5), and still failed 4 of 5 without the second registration.
  • Measured with the fix (Node 22): 12M rows in 60 parts then process.exit(0), 6/6 clean (previously 6/6 crashed), and 3/3 after an engine restart (previously 3/5 crashed). With a 30M-row insert still running, 4/4 clean (previously 3/3 crashed). Under Deno, Deno.exit() and process.exit() with merges in flight are clean (both crashed before). Bun was clean before and after.
  • Not changed, and already the case before this PR: a worker thread that loads chdb runs the addon's env cleanup hook when it exits, and that hook closes every connection in the process, the main thread's included.

Verified locally (Linux x86_64, against rc.2)

  • linux-x86_64-libchdb.tar.gz SHA256 matches the release digest f180eb9d…19ab739a and SHA256SUMS. The other three platform tarballs' release digests match SHA256SUMS. npm run libchdb fetched the same tarball. chdb.h says CHDB_VERSION "26.11.0-rc.2", and chdb_version() through the addon returns 26.11.0-rc.2.
  • npm run build (node-gyp, Python 3.11, no compiler warnings), with no @chdb/lib-* in node_modules, as CI leaves it.
  • npm run test:all, run on Node 18.20.8, 20.20.1 and 22.23.4, with a rebuild per version: mocha 24 passing; vitest 43 files, 488 passed, 11 skipped (one more than before: the new exit case). With the exit fix, on Node 22 and 18: vitest 490 passed, 11 skipped (the two exit cases added). On the seventh commit, Node 22 and 18: vitest 491 passed, 11 skipped (the worker case added), and on Node 22 test:durable:node 24 passed.
  • npm run test:durable (Node 20 and 22): 141 passed, 11 skipped (the S3 file without a bucket). Again on Node 22 with the exit fix: same.
  • npm run test:durable:node (Node 20 and 22): 24 passed. Again on Node 22 with the exit fix: same.
  • npm run test:durable:s3 against the pinned Silo image from CI: 11 passed.
  • Parity runner (client-1.23.0), with this PR's skip list: 175 passed, 36 skipped, 0 failed.
  • Cleanroom path: scripts/build-addon-portable.sh (AlmaLinux 8 re-link, runtime baseline OK), build:platform with CHDB_LIB_VERSION=26.11.0-rc.2.1, npm pack of main and subpackage, install into an empty project, assert-relocatable.sh OK, smoke-e2e.mjs with the build-tree libchdb.so removed, then the same smoke in node:22-bookworm. All OK.
  • Runtimes: the Bun 1.4.4 smoke and the Deno 2.9.8 smoke both OK, before and after the exit fix.
  • Run by hand, not in CI: test:durable:e2e under Bun, 19 passed. test:durable:cross-version with rc.2 as NEW and rc.1, v26.9.0 or v26.7.3 as OLD: all claims hold (the new engine restores the old base and WAL, and the old engine is refused once min_reader rises).
  • Exit paths, by hand, small MergeTree tables: MergeTree on the default connection or in a session, the session open or closed at exit, path switches, two connections, a query still running, a durable object open or closed; ending naturally, with process.exit() or with an uncaught exception. 22 runs before the exit fix, and 34 + 19 + 16 + 22 after it (the last 22 on the seventh commit). Every run had the expected exit code and an empty stderr.

Not verified locally

  • linux-aarch64, macos-14 (arm64) and macos-15-intel: CI only.
  • test:durable:cross-binding (needs Python chdb and chdb-go built against this rc).
  • package-lock.json is unchanged, as in the bot's bumps and Move the engine pin to v26.7.3 #94. Its root optionalDependencies still say 26.7.2-rc.2.1, which was already stale before this PR, and CI uses npm install, not npm ci.

Worth knowing

  • The exit-time stderr from rc.1 is gone. With rc.1, a Node process that used a MergeTree table printed chDB global pool shutdown refused: active jobs now=5; live local pool workers=[MergeTreeBackgroundExecutorThreadsx5] / chDB shutdown failed: global thread pool did not become idle at exit: chdb_shutdown() refused while the MergeTree background executors were alive. rc.2 destroys them on shutdown and also stops jemalloc's background threads (chdb-core#312). The new engine-exit.test.ts case fails on rc.1 with exactly those two lines and passes on rc.2. The addon already calls chdb_shutdown() at exit, so it needed no change.
  • Pre-existing and left alone: the design doc and chdb-node-test.yml still say the newest subpackage on npm is 26.7.0-stable.1, but npm now has 26.7.3-stable.1, pinned by chdb@3.4.0.

🤖 Generated with Claude Code

Note

Pin chdb-core to v26.11.0-rc.2 and add process-exit engine teardown

  • Pins the engine to v26.11.0-rc.2 and native packages to 26.11.0-rc.2.1 in package.json and update_libchdb.sh, with matching README and design doc updates
  • Adds an ensureExitHook teardown in index.js that runs at module init and closes tracked sessions before calling the addon's new ExitTeardown on process exit or Deno unload
  • Adds ExitTeardownWrapper and a live-environment count in lib/chdb_node.cpp. It closes eligible connections and shuts down the engine once; if another addon environment or in-flight connection work exists, it defers cleanup to the atexit fallback
  • Adds four spawned-process teardown tests in test/v3/engine-exit.test.ts covering MergeTree use, default-connection merges, in-flight async inserts, and worker-thread sessions
  • Behavioral Change: shutdown_engine_once() in lib/chdb_node.cpp makes chdb_shutdown() run at most once across exit paths; exit cleanup is now installed at module load rather than on first session

Macroscope summarized e4947ee.

Pin chdb-core v26.11.0-rc.1 (ClickHouse master def2fdf4caa, 26.11.1.1)
to validate the release candidate before the full v26.11.1 release. This
is the same edit the engine release check makes: CHDB_ENGINE_PIN, and
LIBCHDB_NPM_VERSION plus the four @chdb/lib-* optionalDependencies moved
to 26.11.0-rc.1.1, the scheme's first packaging of an rc engine. The
publish workflow already routes an -rc. subpackage to the rc dist-tag.

The places that name the pin in prose move with it: the README's durable
section, the test workflow header and the design doc's still-to-do item.
The update_libchdb.sh comment still described 26.7.2-rc.2 and the old
exact-match rule; it now names this build's chdb_version() string and
the min_reader floor that replaced exact matching.

The C ABI change since v26.7.3 is additive (chdb_insert_arrow_array and
chdb_insert_arrow_stream, unused here) plus documentation, so the addon
needs no change.
The parity runner runs @clickhouse/client 1.23.0's integration suite,
which predates two server changes that every chdb-core engine from
v26.9.0 on carries (v26.7.3 passes all four):

- Analyzer errors now read "identifier `x`. In scope ..." with a capital
  I, so the tag's case-sensitive "in scope" regex misses. Code and type
  still parse correctly.
- An epoch-milliseconds JSON number inserted into DateTime64(3)
  saturates to 9999-12-31 23:59:59.000. A JSON string still parses, and
  chdb-node's own serializers always send Date values as strings.

ClickHouse/clickhouse-js#1025 (in client-1.24.0) relaxed the regex and
skips the DateTime64 cases on servers that clamp. Moving the runner to
1.24.x is a separate sync: it fixes these four but brings three failures
of its own that fail on v26.7.3 as well. Until then, skip these with the
reason recorded, and drop them when syncedAgainst moves.
chdb-core v26.11.0-rc.2 is rc.1 plus chdb-core#312, on the same ClickHouse
baseline (26.11.1.1). For this binding the visible change is at exit:
chdb_shutdown() no longer refuses once a MergeTree table was used, and it
now also stops jemalloc's background threads. chdb.h differs from rc.1
only in CHDB_VERSION, so the addon needs no change.

This is the same edit the engine release check makes: CHDB_ENGINE_PIN,
and LIBCHDB_NPM_VERSION plus the four @chdb/lib-* optionalDependencies
at 26.11.0-rc.2.1, the scheme's first packaging of this engine. The prose
that names the pin moves with it: the README's durable section, the test
workflow header and the design doc's still-to-do item. The update_libchdb.sh
comment names this build's chdb_version() string, and rc.1 joins the
engines that an object written on rc.2 refuses.
On v26.11.0-rc.1 every Node process that had used a MergeTree table
printed "chDB global pool shutdown refused: ..." and "chDB shutdown
failed: global thread pool did not become idle" at exit: chdb_shutdown()
refused while the MergeTree background executors were still alive. The
exit code stayed 0, and the existing exit test uses no MergeTree, so
nothing here noticed. rc.2 fixes it in the engine (chdb-core#312).

The new case leaves a MergeTree session open for the exit hook to close
and requires an empty stderr. Against rc.1 it fails with exactly those
two lines; against rc.2 it passes.
@wudidapaopao wudidapaopao changed the title Pin chdb-core v26.11.0-rc.1 Pin chdb-core v26.11.0-rc.2 Nov 5, 2026
A process.exit() with MergeTree merges in flight on the standalone
query() connection crashed in exit() (SIGSEGV, or "mutex lock failed:
Invalid argument"), every time on v26.11.0-rc.2, after chDB's crash
handler waited about 30 s. The 'exit' hook closed only Sessions, and Node
runs no environment cleanup hooks on process.exit(), so the default
connection was first closed by the addon's atexit backstop. That handler
is registered at the first connect, and atexit handlers run in reverse
order of registration: by then every static constructed later, including
the ones queries and merges construct, had been destroyed, and the server
teardown ran onto them.

The 'exit' hook is now installed when index.js loads, and after closing
the sessions it calls the new native ExitTeardown, which closes every
connection no async call is running on, then calls chdb_shutdown(). This
runs before exit(), with every static still alive. Only the main thread
calls it: the connections and the engine are process-wide, so a worker
that exits must not close them under the main thread. Deno.exit() skips
the 'exit' event and dispatches 'unload', so the hook listens for that
too under Deno.

A connection with a call still running is left open, since closing it
under the call aborts the engine and Node joins that libuv thread only
after the 'exit' event. For that case the backstop is registered again,
so exit() runs it first: after the join and before the destroyed statics.
The backstop stays as the fallback. Its comment said it ran before the
engine's own exit handlers. Since rc.2 the engine registers its teardown
on every start, so after a restart that teardown runs first, and the
backstop also runs after every static constructed after the first
connect. The env cleanup hook's comment no longer calls it the most
reliable cleanup, since it does not run on process.exit().

An 'exit' listener that runs after this hook now finds the engine
closed, as it already found its sessions.

Two tests in engine-exit.test.ts: process.exit() with merges in flight
on the default connection, and with an insert still running on it. Both
fail without this change, the first by crashing, the second by crashing
or hanging until killed.
Every macOS job failed at process exit after the previous commit, with
"libc++abi: terminating due to uncaught exception of type
std::__1::system_error: mutex lock failed: Invalid argument" and an
abort. chdb_shutdown() keeps a function-local static mutex that its first
call constructs. ExitTeardown now makes that first call from the 'exit'
event, after the atexit backstop was registered, so the mutex's
destructor runs before the backstop, and the backstop's own
chdb_shutdown() then locked a destroyed mutex. glibc lets that through,
which is why Linux passed; macOS returns EINVAL and std::mutex throws.
The busy branch hit the same thing the other way round: the backstop
registered again from the hook calls it first, and the original backstop
called it a second time.

The exit paths now share a flag and call chdb_shutdown() once between
them. gdb on Linux shows a single call at exit in both cases, where there
were two before.
Review found a regression in the exit teardown. ExitTeardown, called from
the main thread's 'exit' event, closed every connection whose in-flight
count was 0. Synchronous calls are not in that count, and Node stops
worker threads only after the 'exit' event. A worker looping
`new Session()` and `s.query()` while the main thread called
process.exit(0) therefore ran its next query on a freed connection:
SIGSEGV in chdb_query_n from QueryWithConnection, 3 times out of 3. The
addon on main was clean, because only the atexit backstop closed
connections, after Node had stopped the workers.

The addon now counts the environments that loaded it: Init counts one in
and the env cleanup hook counts it out. While another one is alive,
ExitTeardown closes nothing and registers the backstop again, as it
already does for a connection with an async call running. By the time
exit() runs that backstop, Node has stopped and joined every worker.

New engine-exit.test.ts case: a worker runs synchronous Session queries
in a loop, and the main thread calls process.exit(0). It fails on the
previous commit and passes with this one, as does the reviewer's repro
for both a worker Session and the default connection.
@wudidapaopao
wudidapaopao merged commit 8713f7a into chdb-io:main Nov 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant