Repository navigation
Pin chdb-core v26.11.0-rc.2 - #99
Merged
wudidapaopao merged 7 commits intoNov 5, 2026
Merged
Conversation
Pin chdb-core v26.11.0-rc.1 (ClickHouse master def2fdf4caa, 26.11.1.1) to validate the release candidate before the full v26.11.1 release. This is the same edit the engine release check makes: CHDB_ENGINE_PIN, and LIBCHDB_NPM_VERSION plus the four @chdb/lib-* optionalDependencies moved to 26.11.0-rc.1.1, the scheme's first packaging of an rc engine. The publish workflow already routes an -rc. subpackage to the rc dist-tag. The places that name the pin in prose move with it: the README's durable section, the test workflow header and the design doc's still-to-do item. The update_libchdb.sh comment still described 26.7.2-rc.2 and the old exact-match rule; it now names this build's chdb_version() string and the min_reader floor that replaced exact matching. The C ABI change since v26.7.3 is additive (chdb_insert_arrow_array and chdb_insert_arrow_stream, unused here) plus documentation, so the addon needs no change.
The parity runner runs @clickhouse/client 1.23.0's integration suite, which predates two server changes that every chdb-core engine from v26.9.0 on carries (v26.7.3 passes all four): - Analyzer errors now read "identifier `x`. In scope ..." with a capital I, so the tag's case-sensitive "in scope" regex misses. Code and type still parse correctly. - An epoch-milliseconds JSON number inserted into DateTime64(3) saturates to 9999-12-31 23:59:59.000. A JSON string still parses, and chdb-node's own serializers always send Date values as strings. ClickHouse/clickhouse-js#1025 (in client-1.24.0) relaxed the regex and skips the DateTime64 cases on servers that clamp. Moving the runner to 1.24.x is a separate sync: it fixes these four but brings three failures of its own that fail on v26.7.3 as well. Until then, skip these with the reason recorded, and drop them when syncedAgainst moves.
chdb-core v26.11.0-rc.2 is rc.1 plus chdb-core#312, on the same ClickHouse baseline (26.11.1.1). For this binding the visible change is at exit: chdb_shutdown() no longer refuses once a MergeTree table was used, and it now also stops jemalloc's background threads. chdb.h differs from rc.1 only in CHDB_VERSION, so the addon needs no change. This is the same edit the engine release check makes: CHDB_ENGINE_PIN, and LIBCHDB_NPM_VERSION plus the four @chdb/lib-* optionalDependencies at 26.11.0-rc.2.1, the scheme's first packaging of this engine. The prose that names the pin moves with it: the README's durable section, the test workflow header and the design doc's still-to-do item. The update_libchdb.sh comment names this build's chdb_version() string, and rc.1 joins the engines that an object written on rc.2 refuses.
On v26.11.0-rc.1 every Node process that had used a MergeTree table printed "chDB global pool shutdown refused: ..." and "chDB shutdown failed: global thread pool did not become idle" at exit: chdb_shutdown() refused while the MergeTree background executors were still alive. The exit code stayed 0, and the existing exit test uses no MergeTree, so nothing here noticed. rc.2 fixes it in the engine (chdb-core#312). The new case leaves a MergeTree session open for the exit hook to close and requires an empty stderr. Against rc.1 it fails with exactly those two lines; against rc.2 it passes.
A process.exit() with MergeTree merges in flight on the standalone query() connection crashed in exit() (SIGSEGV, or "mutex lock failed: Invalid argument"), every time on v26.11.0-rc.2, after chDB's crash handler waited about 30 s. The 'exit' hook closed only Sessions, and Node runs no environment cleanup hooks on process.exit(), so the default connection was first closed by the addon's atexit backstop. That handler is registered at the first connect, and atexit handlers run in reverse order of registration: by then every static constructed later, including the ones queries and merges construct, had been destroyed, and the server teardown ran onto them. The 'exit' hook is now installed when index.js loads, and after closing the sessions it calls the new native ExitTeardown, which closes every connection no async call is running on, then calls chdb_shutdown(). This runs before exit(), with every static still alive. Only the main thread calls it: the connections and the engine are process-wide, so a worker that exits must not close them under the main thread. Deno.exit() skips the 'exit' event and dispatches 'unload', so the hook listens for that too under Deno. A connection with a call still running is left open, since closing it under the call aborts the engine and Node joins that libuv thread only after the 'exit' event. For that case the backstop is registered again, so exit() runs it first: after the join and before the destroyed statics. The backstop stays as the fallback. Its comment said it ran before the engine's own exit handlers. Since rc.2 the engine registers its teardown on every start, so after a restart that teardown runs first, and the backstop also runs after every static constructed after the first connect. The env cleanup hook's comment no longer calls it the most reliable cleanup, since it does not run on process.exit(). An 'exit' listener that runs after this hook now finds the engine closed, as it already found its sessions. Two tests in engine-exit.test.ts: process.exit() with merges in flight on the default connection, and with an insert still running on it. Both fail without this change, the first by crashing, the second by crashing or hanging until killed.
Every macOS job failed at process exit after the previous commit, with "libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument" and an abort. chdb_shutdown() keeps a function-local static mutex that its first call constructs. ExitTeardown now makes that first call from the 'exit' event, after the atexit backstop was registered, so the mutex's destructor runs before the backstop, and the backstop's own chdb_shutdown() then locked a destroyed mutex. glibc lets that through, which is why Linux passed; macOS returns EINVAL and std::mutex throws. The busy branch hit the same thing the other way round: the backstop registered again from the hook calls it first, and the original backstop called it a second time. The exit paths now share a flag and call chdb_shutdown() once between them. gdb on Linux shows a single call at exit in both cases, where there were two before.
Review found a regression in the exit teardown. ExitTeardown, called from the main thread's 'exit' event, closed every connection whose in-flight count was 0. Synchronous calls are not in that count, and Node stops worker threads only after the 'exit' event. A worker looping `new Session()` and `s.query()` while the main thread called process.exit(0) therefore ran its next query on a freed connection: SIGSEGV in chdb_query_n from QueryWithConnection, 3 times out of 3. The addon on main was clean, because only the atexit backstop closed connections, after Node had stopped the workers. The addon now counts the environments that loaded it: Init counts one in and the env cleanup hook counts it out. While another one is alive, ExitTeardown closes nothing and registers the backstop again, as it already does for a connection with an async call running. By the time exit() runs that backstop, Node has stopped and joined every worker. New engine-exit.test.ts case: a worker runs synchronous Session queries in a loop, and the main thread calls process.exit(0). It fails on the previous commit and passes with this one, as does the reviewer's repro for both a worker Session and the default connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the engine pin to chdb-core v26.11.0-rc.2 so the release candidate gets validated here. rc.2 is rc.1 plus chdb-core#312, on the same ClickHouse baseline (master
def2fdf4caa, 26.11.1.1). It is a release candidate: it is not on PyPI, and the full release will be v26.11.1. chdb-core'scheck_bindings.ymlonly dispatches the engine release check for full releases, so this PR does that check's edit by hand. The PR first pinned rc.1; the third commit moves it to rc.2. The fifth to seventh fix an exit crash on the standalone connection (below).What changed
update_libchdb.sh:CHDB_ENGINE_PIN=v26.11.0-rc.2.LIBCHDB_NPM_VERSIONand the four@chdb/lib-*optionalDependenciesinpackage.jsonbecome26.11.0-rc.2.1. That is the scheme's value for a first packaging of an rc engine, andscripts/check-version-scheme.mjspasses. These are the linesengine-release-check.yml'spropose_bumprewrites. The publish workflow already sends an-rc.subpackage to thercdist-tag, so nothing else changes for an rc.chdb-node-test.yml, and the still-to-do item indocs/design/durable-control-plane.md. Theupdate_libchdb.shcomment gives this build'schdb_version()string (26.11.0-rc.2) and themin_readerfloor that replaced exact matching: an object written on rc.2 is refused by rc.1 and earlier.tests/clickhouse-js/skip_list.json: four parity cases skipped, each with its reason (details below).test/v3/engine-exit.test.ts: a new case. A process that used a MergeTree table and exits with its session still open must leave stderr empty. It fails on rc.1 (see below) and passes on rc.2.index.js,lib/chdb_node.cpp, two moreengine-exit.test.tscases): the process'exit'hook now also closes the standalone default connection and stops the engine before Cexit(). Details below.The pin needs no addon change: rc.2's
chdb.hdiffers from rc.1's only inCHDB_VERSION. Between v26.7.3 and rc.1,chdb.hgainedchdb_insert_arrow_array/chdb_insert_arrow_stream, which the addon doesn't use, plus doc comments.Parity skips (second commit)
The
connection / parityjob runs@clickhouse/client1.23.0's suite. Four of its cases fail on every engine from v26.9.0 on. I ran the runner against v26.7.3 (all pass), v26.9.0, rc.1 and rc.2 (the same 4 fail on each):error_parsing"unknown identifier" / "unknown table": the analyzer now writes. In scopewith a capital I, and the 1.23.0 regex is case-sensitive. Code and type still parse correctly.date_time"DateTime64(3) … (different timezone)" anddata_types"custom JSON handling (BigInt and Date)": a JSON number of epoch milliseconds inserted intoDateTime64(3)now saturates to9999-12-31 23:59:59.000. I confirmed this with plain SQL (format(JSONEachRow, 'd DateTime64(3)', '{"d":1662328969123}')). The same value sent as a JSON string still parses. chdb-node's own serializers sendDateas a string, so this only affects callers who put raw epoch-ms numbers into JSONEachRow themselves.Upstream fixed both in ClickHouse/clickhouse-js#1025 (released in client-1.24.0): it relaxed the regex and skips the DateTime64 cases on servers that clamp. I tried moving the runner to
client-1.24.1. That fixes these four but brings three other failures (should not destroy the socket under the next request after the idle TTL, and twoArray(Date)query-binding cases), and those also fail on v26.7.3, so they are not caused by this engine. That sync belongs in its own PR. Each skip entry says to drop it whensyncedAgainstmoves.Exit fix (fifth to seventh commits)
process.exit()with MergeTree merges in flight on the standalonequery()connection crashed inexit(): SIGSEGV, ormutex lock failed: Invalid argument, after chDB's crash handler waited about 30 s. On rc.2 that happened 6 times out of 6. It is not new in rc.2; rc.1 hung instead. The JS'exit'hook closed onlySessions, and Node runs no environment cleanup hooks onprocess.exit(). So the default connection was first closed by the addon'sstd::atexitbackstop. That handler is registered at the first connect, and atexit handlers run in reverse order of registration, so by then every static constructed later, including the ones queries and merges construct, had been destroyed. gdb shows the crash inchdb_close_conncalled fromshutdown_on_process_exit.'exit'hook is installed whenindex.jsloads, not with the firstSession. After closing the sessions it calls a new nativeExitTeardown. That closes every connection no async call is running on (the addon's existing in-flight count) and then callschdb_shutdown(), all beforeexit()while every static is alive. Main thread only: the connections and the engine are process-wide, so a worker that exits must not close them under the main thread.Deno.exit()skips the'exit'event and dispatchesunload, so under Deno the hook listens for that too.'exit'event. For that case the backstop is registered again, soexit()runs it first: after the join, before the destroyed statics.chdb_shutdown()is called at most once between the exit paths (sixth commit). It keeps a function-local static mutex that its first call constructs. Called first from the'exit'event, that mutex is destroyed before the atexit backstop runs, and a second call from the backstop locked the destroyed mutex. glibc lets that through. macOS aborts withmutex lock failed: Invalid argument, which failed every macOS job on the fifth commit. gdb now shows one call at exit, where there were two.worker_threadsWorker has the addon loaded, the'exit'teardown closes nothing (seventh commit). A worker may be using any connection, and synchronous calls are not in the in-flight count. Node stops workers only after the'exit'event, so the fifth commit's teardown freed a worker'sSessionunder its nexts.query(), a SIGSEGV found in review. The addon now counts the environments that loaded it. While more than one is alive, it only registers the backstop again, the same as for a busy connection. By the timeexit()runs it, Node has stopped and joined the workers. A newengine-exit.test.tscase runs synchronousSessionqueries in a worker while the main thread callsprocess.exit(0). It fails on the sixth commit and passes on the seventh.'exit'listeners. The hook is installed whenrequire('chdb')runs. Anyprocess.on('exit')listener added after that runs after the hook, and now finds the engine stopped: a standalonequery()there throwsFailed to acquire default connection, and a newSessionfails too. Such listeners already found their sessions closed. Listeners added beforerequire('chdb')run first and are not affected.engine-exit.test.tscases in the fifth commit:process.exit()with merges in flight on the default connection, and with a 10M-row insert still running on it. Both check that merges were really running (and the insert really unfinished) when the exit began. Both fail without the fix. The first crashes; the second crashed or hung until killed (5 of 5), and still failed 4 of 5 without the second registration.process.exit(0), 6/6 clean (previously 6/6 crashed), and 3/3 after an engine restart (previously 3/5 crashed). With a 30M-row insert still running, 4/4 clean (previously 3/3 crashed). Under Deno,Deno.exit()andprocess.exit()with merges in flight are clean (both crashed before). Bun was clean before and after.Verified locally (Linux x86_64, against rc.2)
linux-x86_64-libchdb.tar.gzSHA256 matches the release digestf180eb9d…19ab739aand SHA256SUMS. The other three platform tarballs' release digests match SHA256SUMS.npm run libchdbfetched the same tarball.chdb.hsaysCHDB_VERSION "26.11.0-rc.2", andchdb_version()through the addon returns26.11.0-rc.2.npm run build(node-gyp, Python 3.11, no compiler warnings), with no@chdb/lib-*innode_modules, as CI leaves it.npm run test:all, run on Node 18.20.8, 20.20.1 and 22.23.4, with a rebuild per version: mocha 24 passing; vitest 43 files, 488 passed, 11 skipped (one more than before: the new exit case). With the exit fix, on Node 22 and 18: vitest 490 passed, 11 skipped (the two exit cases added). On the seventh commit, Node 22 and 18: vitest 491 passed, 11 skipped (the worker case added), and on Node 22test:durable:node24 passed.npm run test:durable(Node 20 and 22): 141 passed, 11 skipped (the S3 file without a bucket). Again on Node 22 with the exit fix: same.npm run test:durable:node(Node 20 and 22): 24 passed. Again on Node 22 with the exit fix: same.npm run test:durable:s3against the pinned Silo image from CI: 11 passed.client-1.23.0), with this PR's skip list: 175 passed, 36 skipped, 0 failed.scripts/build-addon-portable.sh(AlmaLinux 8 re-link, runtime baseline OK),build:platformwithCHDB_LIB_VERSION=26.11.0-rc.2.1,npm packof main and subpackage, install into an empty project,assert-relocatable.shOK,smoke-e2e.mjswith the build-treelibchdb.soremoved, then the same smoke innode:22-bookworm. All OK.test:durable:e2eunder Bun, 19 passed.test:durable:cross-versionwith rc.2 as NEW and rc.1, v26.9.0 or v26.7.3 as OLD: all claims hold (the new engine restores the old base and WAL, and the old engine is refused oncemin_readerrises).process.exit()or with an uncaught exception. 22 runs before the exit fix, and 34 + 19 + 16 + 22 after it (the last 22 on the seventh commit). Every run had the expected exit code and an empty stderr.Not verified locally
test:durable:cross-binding(needs Python chdb and chdb-go built against this rc).package-lock.jsonis unchanged, as in the bot's bumps and Move the engine pin to v26.7.3 #94. Its rootoptionalDependenciesstill say26.7.2-rc.2.1, which was already stale before this PR, and CI usesnpm install, notnpm ci.Worth knowing
chDB global pool shutdown refused: active jobs now=5; live local pool workers=[MergeTreeBackgroundExecutorThreadsx5]/chDB shutdown failed: global thread pool did not become idleat exit:chdb_shutdown()refused while the MergeTree background executors were alive. rc.2 destroys them on shutdown and also stops jemalloc's background threads (chdb-core#312). The newengine-exit.test.tscase fails on rc.1 with exactly those two lines and passes on rc.2. The addon already callschdb_shutdown()at exit, so it needed no change.chdb-node-test.ymlstill say the newest subpackage on npm is26.7.0-stable.1, but npm now has26.7.3-stable.1, pinned bychdb@3.4.0.🤖 Generated with Claude Code
Note
Pin chdb-core to v26.11.0-rc.2 and add process-exit engine teardown
ensureExitHookteardown in index.js that runs at module init and closes tracked sessions before calling the addon's newExitTeardownon process exit or Deno unloadExitTeardownWrapperand a live-environment count in lib/chdb_node.cpp. It closes eligible connections and shuts down the engine once; if another addon environment or in-flight connection work exists, it defers cleanup to theatexitfallbackshutdown_engine_once()in lib/chdb_node.cpp makeschdb_shutdown()run at most once across exit paths; exit cleanup is now installed at module load rather than on first sessionMacroscope summarized e4947ee.