From 271ef161ea4c67c9537eefd3f425b3adfb868236 Mon Sep 17 00:00:00 2001 From: Shawn Chen Date: Wed, 16 Sep 2026 10:07:25 +1200 Subject: [PATCH 1/3] Take the other bindings' spelling of the local backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while verifying interop before a release, against Python `chdb` 4.4.0. **`local:` was refused.** Python spells the local backend `local:`, Go registers the same backend under both names, and its comment says why: a namespace URL is the sort of thing that ends up in a config file shared between a Python service and a Node one. This binding took only `file:`, which made it the one place a working configuration stopped working. It now registers both. The two forms differ in how a path is written, not where it points — `file:` is a real URL so a space is `%20`, `local:` is opaque so the path is closer to literal. My first plan was to mirror Python exactly and leave `local:` undecoded. Reading Go killed that: Go decodes both, so Python and Go already disagree past ordinary paths and there is no three-way behaviour to match. Following Go is one factory instead of two parsers, and it is the shape the existing `registerBackendScheme` already wanted. `local://data/objects` is refused rather than silently resolved to `/objects`, and a stray `%` reports which URL was malformed instead of surfacing a bare `URIError`. **The AWS SDK is now loaded on first use.** It is an optional peer dependency, so "I do want S3 and forgot to install it" is a real path — and a static import made that a bare `Cannot find module '@aws-sdk/client-s3'` thrown from `require('chdb/durable/s3')`, a file the caller has never heard of. Python answers the same mistake with `MissingDependency: pip install 'chdb[durable]'`. Deferring the load moves the failure to the first S3 operation and lets it name the package; the scheme registration, key validation and size limit all still work without it. Two verifications are now scripts rather than notes, both run by hand: - `test:durable:cross-version` — a newer engine restores an older one's base and WAL, and the raised floor then refuses the older engine. This could not be tested until `26.7.3` shipped, since `26.7.2-rc.2` was the only Durable-capable engine. It re-execs per stage, which is load-bearing: a libchdb is `dlopen`ed once per process, so a single-process version loads the first library and answers every later request with it — both versions report the same number and it passes while proving nothing. I wrote that version first and it did exactly that. - `test:durable:cross-binding` — Python writes a base plus WAL and Node restores it; Node appends and Python reads it back; Node checkpoints and Python RESTOREs from that archive with no WAL left; and one `local:` URL serves both. `destroy`/`scan` are deliberately still absent. Python has them, but the contract removed `delete_prefix`/destroy from V1 and deferred GC to its own safety design (§8.1 item 4), and Go does not have them either. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 4 + docs/design/durable-control-plane.md | 82 ++++++++++-- package.json | 4 +- scripts/durable-cross-binding.mjs | 193 +++++++++++++++++++++++++++ scripts/durable-cross-version.ts | 139 +++++++++++++++++++ src/durable/backends/s3.ts | 60 +++++++-- src/durable/namespace.ts | 52 +++++++- test/durable/protocol.test.ts | 54 ++++++++ test/durable/subpath.test.ts | 36 +++++ 9 files changed, 595 insertions(+), 29 deletions(-) create mode 100644 scripts/durable-cross-binding.mjs create mode 100644 scripts/durable-cross-version.ts diff --git a/README.md b/README.md index 024ce0b..98358e3 100644 --- a/README.md +++ b/README.md @@ -226,6 +226,10 @@ object needs a private one, so a process holds one open durable object at a time and no ordinary `Session` beside it — fan-out goes across worker processes. +The local backend answers to both `file:` and `local:`, because a namespace URL +gets shared between services in different languages and Python spells it +`local:` while Go takes either. + Recovery on another machine needs the object to live somewhere neither machine owns, so `chdb/durable/s3` provides an S3-compatible backend — AWS S3, Cloudflare R2 and MinIO through one implementation. It sits behind its own diff --git a/docs/design/durable-control-plane.md b/docs/design/durable-control-plane.md index 84e7119..dc886a9 100644 --- a/docs/design/durable-control-plane.md +++ b/docs/design/durable-control-plane.md @@ -266,6 +266,19 @@ can plant it can rewrite the objects directly — and check-then-use cannot be made atomic without `openat`, which Node does not expose. Anything wanting that property should not be on this backend. +It answers to two scheme names, `file:` and `local:`. That is not a +convenience — a namespace URL ends up in a config file, and that config file +gets shared between services in different languages. Python spells this +backend `local:`, Go registers both, and this binding took only `file:` until +it was pointed out, which made Node the one place a working configuration +stopped working. The two forms differ in how a path is written rather than +where it points: `file:` is a real URL, so a space is `%20`; `local:` is +opaque, so the path is closer to literal. Past ordinary paths the bindings do +not agree anyway — Python leaves `local:` undecoded, Go decodes it — so this +follows Go rather than inventing a third behaviour for a case none of them +share. A `local:` URL naming a host (`local://data/objects`) is refused +instead of silently resolving to `/objects`. + A local directory cannot be a remote authority. When the machine holding it is gone, so is the object — so the local backend is what conformance and development run on, not what makes a database recoverable somewhere else. @@ -416,6 +429,24 @@ npm run test:durable:e2e # end-to-end against a real libchdb, under Bun npm run test:durable:stack # the whole stack: Bun + libchdb + real object storage ``` +Two more run by hand, because each needs something CI does not have. They +check the promises that only move when core releases, so the cost of running +them at adoption time rather than per-commit is low: + +```sh +CHDB_LIBCHDB_OLD=…/26.7.2-rc.2/libchdb.so CHDB_LIBCHDB_NEW=…/26.7.3/libchdb.so \ + npm run test:durable:cross-version # a newer engine restores an older one's archive + +CHDB_PYTHON=/tmp/pyenv/bin/python \ + npm run test:durable:cross-binding # Python writes, Node reads, and back +``` + +`cross-version` re-execs itself once per stage. A libchdb is `dlopen`ed once +per process — it has to be, since the engine binds one data path per process — +so a single-process version would load the first library and answer every +later request with it: both versions report the same number, the floor never +rises, and the script passes while proving nothing. + The last one is the arrangement a downstream actually deploys, and the only place the other two suites' halves meet. Each of them covers one: a real engine over a local directory, or a real bucket under a fake engine. Both can pass @@ -453,7 +484,7 @@ CHDB_LIBCHDB_PATH=/path/to/chdb-core/buildlib/libchdb.so npm run test:durable:e2 `CHDB_LIBCHDB_PATH` is only needed when a platform package that predates the durable ABI is installed. The resolver prefers `@chdb/lib-*` over anything in the working tree, so a stale one shadows a local `chdb-core` build — and until -the RC platform packages are published, the newest on npm is +a Durable-capable platform package is published, the newest on npm is `26.7.0-stable.1`, which exports none of the durable symbols. Either override the path or `rm -rf node_modules/@chdb/lib-*`, which is what CI already does. Running against a shadowing build fails with a message naming the missing ABI @@ -497,12 +528,37 @@ Against the V1 conformance list, this binding's position: | Single-writer races, fencing, heartbeat, failed and ambiguous commits | covered | | Unknown-field round trips, size limits, secret redaction, close failure | covered | | Full provider suite against a real object store | covered — AWS S3 and MinIO | -| Restoring full backups from *earlier* core releases | not applicable yet — `v26.7.2-rc.2` is the first release with the Durable ABI, so there is no earlier archive to restore | -| New-header/old-library and old-header/new-library ABI tests | not applicable yet — same reason; the promise starts here | -| Every writer's fixture read by two other bindings | pending — needs Python and Go to reach V1 | +| Restoring full backups from *earlier* core releases | covered — `npm run test:durable:cross-version` | +| Old-header/new-library ABI | covered — an addon compiled against `26.7.2-rc.2` headers runs the whole suite on `26.7.3` | +| New-header/old-library ABI | not covered — needs an addon built against headers newer than the engine it loads, which no released pair produces yet | +| Every writer's fixture read by two other bindings | Python covered — `npm run test:durable:cross-binding`; Go pending | + +Two of these became testable when `26.7.3` shipped, since `26.7.2-rc.2` then +stopped being the only Durable-capable engine. Both scripts are run by hand +rather than in CI — one needs two engine downloads, the other a second +language runtime — and what they check moves only when core releases. + +Measured on `26.7.2-rc.2` → `26.7.3`: + +| Claim | Result | +| --- | --- | +| New engine restores a base archive the old engine wrote | holds | +| …and replays a WAL segment the old engine appended | holds | +| Writing raises `min_reader`, and the old engine is then refused `engine_incompatible` | holds | + +Measured against Python `chdb` 4.4.0 (same `26.7.3` engine, its own bindings): + +| Exchange | Result | +| --- | --- | +| Python base + WAL, restored and replayed by Node | holds | +| Node appends to a Python-written object; Python reads it back | holds | +| Node checkpoints; Python RESTOREs from that archive with no WAL left | holds | +| Both on a real object store (MinIO), not just a local directory | holds | +| One `local:` URL string opened by both | holds | -The two "not applicable" rows become real on the second Durable-capable core -release, and the fixtures written now are what they will be tested against. +Go reached V1 as well (`chdb-go/chdb/durable`), so the contract's +read-by-two-other-bindings requirement is one pair short rather than two. The +Go exchange has not been run. ### Provider conformance @@ -533,12 +589,12 @@ durable object written on one "machine" — one namespace instance, one engine, one scratch tree — reopened on another that shares nothing but the bucket, for both a WAL-only object and a checkpointed one. -Run against engine `26.7.2-rc.2`: +Run against engine `26.7.3`: | Provider | Status | | --- | --- | -| AWS S3 (`us-east-2`) | 11/11 conformance, plus 4/4 full stack | -| MinIO (`RELEASE.2025-09-07`) | 11/11 conformance, run repeatedly | +| MinIO (latest) | 11/11 conformance, plus 4/4 full stack, plus the cross-binding exchange | +| AWS S3 (`us-east-2`) | 11/11 conformance and 4/4 full stack on `26.7.2-rc.2`; not re-run on `26.7.3` | | Cloudflare R2 | not run — the code path is the same, but the claim is not made until measured | The two that have run agree on every point, including the ETag behaviour above: @@ -664,5 +720,9 @@ this package: which the loader prefers over a local build, and *that* is the case the adapter refuses by name. Until a tag goes out, using `chdb/durable/node` from a checkout means `npm run build` plus `rm -rf node_modules/@chdb/lib-*`. -2. Add the shared cross-binding fixtures from the chdb repository once they - exist, and read a Python-written object with them. +2. Run the cross-binding exchange against Go as well. Python is covered by + `npm run test:durable:cross-binding`, and `chdb-go/chdb/durable` reached V1 + too, so the contract's read-by-two-other-bindings requirement is one pair + short. The shared fixtures in the chdb repository are still worth adopting + when they exist — a fixture pins a byte layout, while these scripts only + prove two live implementations agree. diff --git a/package.json b/package.json index b644d8e..ad849c9 100644 --- a/package.json +++ b/package.json @@ -77,7 +77,9 @@ "test:durable:node": "npm run build:ts && vitest run --config vitest.durable-node.config.ts", "test:durable:e2e": "bun test --timeout 120000 test/durable/libchdb-e2e.bun.test.ts", "test:durable:s3": "vitest run --config vitest.durable.config.ts test/durable/s3-backend.test.ts", - "test:durable:stack": "bun test --timeout 300000 test/durable/full-stack.bun.test.ts" + "test:durable:stack": "bun test --timeout 300000 test/durable/full-stack.bun.test.ts", + "test:durable:cross-version": "bun scripts/durable-cross-version.ts", + "test:durable:cross-binding": "npm run build:ts && node scripts/durable-cross-binding.mjs" }, "author": { "name": "chdb", diff --git a/scripts/durable-cross-binding.mjs b/scripts/durable-cross-binding.mjs new file mode 100644 index 0000000..35aaf0c --- /dev/null +++ b/scripts/durable-cross-binding.mjs @@ -0,0 +1,193 @@ +#!/usr/bin/env node +/** + * Can another binding read what this one wrote, and the other way round? + * + * The V1 contract's cross-binding requirement is that a writer's object be + * readable by the other implementations — the on-disk format is the interface, + * not any one library's API. Python (`chdb` ≥ 4.4.0) and Go both reached V1, + * so this pairs Node against Python; Go is not covered here. + * + * Four exchanges, because each proves something the others do not: + * + * 1. Python writes a base archive plus a WAL segment; Node opens it. Proves + * Node can RESTORE a foreign archive and replay a foreign WAL. + * 2. Node appends to that same object; Python reads it back. Proves a + * foreign writer's head/CAS/lease round trip is accepted. + * 3. Node creates its own object and checkpoints; Python restores purely + * from Node's base, with no WAL left. The strongest direction: Python's + * engine RESTOREs an archive Node's BACKUP produced. + * 4. Both spellings of the local backend URL name the same object — Python + * writes `local:`, Node reads the identical string. + * + * ```sh + * python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0' + * npm run build # the addon must carry the durable ABI + * CHDB_PYTHON=/tmp/pyenv/bin/python node scripts/durable-cross-binding.mjs + * ``` + * + * Not in CI: it needs a second language runtime with its own engine build. + * Run it when adopting an engine, and record the result in + * docs/design/durable-control-plane.md. + */ + +import { execFileSync } from 'child_process' +import { mkdtempSync, writeFileSync, rmSync } from 'fs' +import { tmpdir } from 'os' +import { join } from 'path' +import { createRequire } from 'module' + +const require = createRequire(join(process.cwd(), 'package.json')) +const PYTHON = process.env.CHDB_PYTHON ?? 'python3' + +function pythonHasDurable() { + try { + const v = execFileSync(PYTHON, ['-c', 'import chdb,chdb.durable;print(chdb.__version__)'], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim() + return v + } catch { + return null + } +} + +const pyVersion = pythonHasDurable() +if (!pyVersion) { + console.error( + `${PYTHON} has no chdb.durable. Install it:\n` + + ` python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0'\n` + + ` CHDB_PYTHON=/tmp/pyenv/bin/python node scripts/durable-cross-binding.mjs`, + ) + process.exit(2) +} + +const { DurableNamespace, nodeEngineFactory } = require('./dist/durable/node.js') + +const root = mkdtempSync(join(tmpdir(), 'durable-xbind-')) +let failed = false + +function report(claim, ok, detail = '') { + console.log(`${ok ? 'PASS' : 'FAIL'} ${claim}${detail ? ` — ${detail}` : ''}`) + if (!ok) failed = true +} + +/** Run a Python snippet against the shared namespace; returns its stdout. */ +function python(body) { + const file = join(root, `py-${Math.random().toString(36).slice(2)}.py`) + writeFileSync(file, body) + return execFileSync(PYTHON, [file], { encoding: 'utf8', env: { ...process.env, XBIND_ROOT: root } }) +} + +/** A namespace on the shared directory, driven by this package's addon. */ +function namespace(url, scratchName) { + return new DurableNamespace(url, { + engineFactory: nodeEngineFactory(), + scratchRoot: mkdtempSync(join(tmpdir(), `${scratchName}-`)), + tuning: { leaseTtlMs: 120_000, heartbeatIntervalMs: 30_000 }, + }) +} + +const NODE_ENGINE = await (async () => { + const { ChdbNodeEngine } = require('./dist/durable/node.js') + return new ChdbNodeEngine().version() +})() +console.log(`node addon engine ${NODE_ENGINE} <-> python chdb ${pyVersion}\n`) + +try { + // 1. Python writes base + WAL. + python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('shared') +o.execute("CREATE TABLE events (id UInt64, note String) ENGINE = MergeTree ORDER BY id") +o.execute("INSERT INTO events VALUES (1, 'py-one'), (2, 'py-two')") +o.checkpoint() +o.execute("INSERT INTO events VALUES (3, 'py-three')") +o.flush() +o.close() +`) + + // 2. Node restores it, then appends. + { + const ns = namespace(`file://${root}/ns`, 'xbind-node') + const phases = [] + const obj = await ns.open('shared', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM events ORDER BY id', { format: 'CSV' }) + report('Node restores a Python base', phases.includes('restoring-base')) + report('Node replays a Python WAL', phases.includes('replaying-wal')) + report( + 'Python -> Node rows match', + rows === '1,"py-one"\n2,"py-two"\n3,"py-three"\n', + rows.trim().replace(/\n/g, ' | '), + ) + await obj.execute("INSERT INTO events VALUES (4, 'node-four')") + await obj.flush() + await obj.close() + } + + // 3. Python reads Node's append. + { + const out = python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('shared') +print(o.query('SELECT id, note FROM events ORDER BY id','CSV').bytes().decode(), end='') +o.close() +`) + report( + 'Node -> Python rows match', + out === '1,"py-one"\n2,"py-two"\n3,"py-three"\n4,"node-four"\n', + out.trim().replace(/\n/g, ' | '), + ) + } + + // 4. Node creates its own object and checkpoints; Python restores from that + // base alone, with no WAL to fall back on. + { + const ns = namespace(`file://${root}/ns`, 'xbind-nw') + const obj = await ns.open('nodeobj', { database: 'default' }) + await obj.execute('CREATE TABLE t (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO t VALUES (1, 'from-node')") + await obj.checkpoint() + report('Node checkpoint left no WAL', obj.manifest.wal.length === 0) + await obj.close() + + const out = python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('nodeobj') +assert len(o.wal) == 0, 'expected a checkpoint-only object' +print(o.query('SELECT id, note FROM t','CSV').bytes().decode(), end='') +o.close() +`) + report("Python RESTOREs Node's archive", out === '1,"from-node"\n', out.trim()) + } + + // 5. One `local:` URL string, both bindings. + { + const url = `local:${root}/shared-url` + python(` +import os +from chdb.durable import Namespace +ns = Namespace('local:' + os.environ['XBIND_ROOT'] + '/shared-url', db='default') +o = ns.open('o') +o.execute("CREATE TABLE u (who String) ENGINE = MergeTree ORDER BY who") +o.execute("INSERT INTO u VALUES ('python')") +o.flush() +o.close() +`) + const ns = namespace(url, 'xbind-url') + const obj = await ns.open('o') + const rows = await obj.query('SELECT who FROM u', { format: 'CSV' }) + report('one local: URL serves both bindings', rows === '"python"\n', rows.trim()) + await obj.close() + } +} finally { + rmSync(root, { recursive: true, force: true }) +} + +console.log(failed ? '\nSOMETHING FAILED' : '\nall exchanges hold') +process.exit(failed ? 1 : 0) diff --git a/scripts/durable-cross-version.ts b/scripts/durable-cross-version.ts new file mode 100644 index 0000000..fdad93d --- /dev/null +++ b/scripts/durable-cross-version.ts @@ -0,0 +1,139 @@ +/** + * Does a newer engine restore what an older one wrote? + * + * That is the compatibility promise `chdb/durable` rests on — the `min_reader` + * and `backup_format` gates exist to admit exactly this case — and until + * `26.7.3` shipped it could not be tested, because `26.7.2-rc.2` was the first + * engine with the durable ABI and there was nothing earlier to restore. + * + * Three claims, in order: + * + * 1. an object written by the OLD engine (full checkpoint plus a WAL segment + * on top) opens on the NEW one and yields the same rows; + * 2. the floor rises on that write — `min_reader` becomes the new version — + * and the OLD engine is then refused with `engine_incompatible` rather + * than left to fail partway through RESTORE; + * 3. neither direction silently half-works. + * + * Runs under Bun because the addon links one engine at build time, while + * `bun:ffi` can `dlopen` whichever it is handed. + * + * Each stage runs in its OWN process, and that is not tidiness. A libchdb is + * `dlopen`ed once per process — `loadLibchdb` caches it, and it has to, + * because the engine binds one data path per process and a second `dlopen` + * would be a second EmbeddedServer. So a single-process version of this + * script silently tests one engine twice: the first library loaded answers + * every later request, both versions report the same number, and the floor + * never rises. It passes while proving nothing. Hence the re-exec below. + * + * ```sh + * CHDB_LIBCHDB_OLD=/path/to/26.7.2-rc.2/libchdb.so \ + * CHDB_LIBCHDB_NEW=/path/to/26.7.3/libchdb.so \ + * bun scripts/durable-cross-version.ts + * ``` + * + * Not in CI: it needs two ~100 MB engine downloads, and what it checks moves + * only when core releases. Run it when adopting an engine, and record the + * result in docs/design/durable-control-plane.md. + */ + +import { mkdtemp, mkdir, rm } from 'fs/promises' +import { tmpdir } from 'os' +import { join } from 'path' + +import { DurableNamespace } from '../src/durable/namespace' +import { isDurableErrorOf } from '../src/durable/errors' +import { LibchdbFfiEngine } from '../test/durable/libchdb-ffi' + +const OLD = process.env['CHDB_LIBCHDB_OLD'] +const NEW = process.env['CHDB_LIBCHDB_NEW'] +if (!OLD || !NEW) { + console.error( + 'Set CHDB_LIBCHDB_OLD and CHDB_LIBCHDB_NEW to two libchdb builds.\n' + + 'Fetch one with: curl -L -o old.tar.gz \\\n' + + ' https://github.com/chdb-io/chdb-core/releases/download/v26.7.2-rc.2/macos-arm64-libchdb.tar.gz', + ) + process.exit(2) +} + +const ROWS = '1,"before-checkpoint"\n2,"also-before"\n3,"after-checkpoint"\n' + +/** One namespace over a shared object store, driven by the given engine. */ +async function namespaceOn(library: string, root: string, scratch: string): Promise { + await mkdir(scratch, { recursive: true }) + return new DurableNamespace(`file://${root}/ns`, { + engineFactory: () => new LibchdbFfiEngine({ libraryPath: library }), + scratchRoot: scratch, + // Long enough that a checkpoint never races the lease on a slow machine. + tuning: { leaseTtlMs: 120_000, heartbeatIntervalMs: 30_000 }, + }) +} + +function report(claim: string, ok: boolean, detail = ''): void { + console.log(`${ok ? 'PASS' : 'FAIL'} ${claim}${detail ? ` — ${detail}` : ''}`) + if (!ok) process.exitCode = 1 +} + +const phase = process.env['XVER_PHASE'] +const root = process.env['XVER_ROOT'] + +if (phase && root) { + // ---- a single stage, in its own process with exactly one engine ---- + if (phase === 'write') { + const ns = await namespaceOn(OLD, root, join(root, 'scratch-old')) + const obj = await ns.open('xver', { database: 'default' }) + await obj.execute('CREATE TABLE events (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO events VALUES (1, 'before-checkpoint'), (2, 'also-before')") + await obj.checkpoint() + await obj.execute("INSERT INTO events VALUES (3, 'after-checkpoint')") + await obj.flush() + await obj.close() + console.log(`WROTE ${await new LibchdbFfiEngine({ libraryPath: OLD }).version()}`) + } else if (phase === 'restore') { + const ns = await namespaceOn(NEW, root, join(root, 'scratch-new')) + const phases: string[] = [] + const obj = await ns.open('xver', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM events ORDER BY id', { format: 'CSV' }) + report('new engine restores the old base', phases.includes('restoring-base')) + report('and replays the WAL on top', phases.includes('replaying-wal')) + report('rows survive the version change', rows === ROWS, rows === ROWS ? '' : JSON.stringify(rows)) + await obj.close() + console.log(`RESTORED ${await new LibchdbFfiEngine({ libraryPath: NEW }).version()}`) + } else if (phase === 'refuse') { + const ns = await namespaceOn(OLD, root, join(root, 'scratch-back')) + const e = await ns.open('xver').then( + (o) => o.close().then(() => undefined), + (err: unknown) => err, + ) + report( + 'old engine refused once the floor rose', + isDurableErrorOf(e, 'engine_incompatible'), + e instanceof Error ? e.message.slice(0, 110) : 'it opened', + ) + } else { + console.error(`unknown XVER_PHASE ${phase}`) + process.exit(2) + } + process.exit(process.exitCode ?? 0) +} + +// ---- orchestrator ---- +const workdir = await mkdtemp(join(tmpdir(), 'durable-xver-')) +let failed = false +try { + console.log(`old: ${OLD}\nnew: ${NEW}\n`) + for (const stage of ['write', 'restore', 'refuse'] as const) { + const proc = Bun.spawnSync({ + cmd: [process.execPath, import.meta.path], + env: { ...process.env, XVER_PHASE: stage, XVER_ROOT: workdir }, + stdout: 'inherit', + stderr: 'inherit', + }) + if (proc.exitCode !== 0) failed = true + } +} finally { + await rm(workdir, { recursive: true, force: true }) +} + +console.log(failed ? '\nSOMETHING FAILED' : '\nall claims hold') +process.exit(failed ? 1 : 0) diff --git a/src/durable/backends/s3.ts b/src/durable/backends/s3.ts index eef87c4..41373fc 100644 --- a/src/durable/backends/s3.ts +++ b/src/durable/backends/s3.ts @@ -72,11 +72,11 @@ import { createReadStream } from 'fs' import { stat } from 'fs/promises' import type { Readable } from 'stream' -import { - GetObjectCommand, - PutObjectCommand, - S3Client, - type S3ClientConfig, +import type { + GetObjectCommand as GetObjectCommandCtor, + PutObjectCommand as PutObjectCommandCtor, + S3Client as S3ClientCtor, + S3ClientConfig, } from '@aws-sdk/client-s3' import type { DurableBackend, GetWithEtag, PutOutcome, ReplaceOutcome } from '../backend' @@ -84,6 +84,40 @@ import { DurableBackendError, DurableLimitExceededError } from '../errors' import { isValidObjectKey } from '../keys' import { registerBackendScheme } from '../namespace' +/** + * The AWS SDK, loaded on first use rather than on import. + * + * `@aws-sdk/client-s3` is an *optional* peer dependency — a caller using only + * the local backend should not have to install several megabytes of it. The + * cost of that is the failure a caller meets when they do want S3 and have not + * installed it, and a static import made that failure a bare + * `Cannot find module '@aws-sdk/client-s3'` thrown from a file they have never + * heard of, at `require('chdb/durable/s3')`. + * + * Deferring the load moves it to the first S3 operation and lets it say what + * to install. The rest of this module — the scheme registration, the key + * validation, the size limit — loads and works either way. + */ +type AwsS3Sdk = typeof import('@aws-sdk/client-s3') + +let sdk: AwsS3Sdk | undefined + +function awsSdk(): AwsS3Sdk { + if (sdk) return sdk + try { + // eslint-disable-next-line @typescript-eslint/no-var-requires + sdk = require('@aws-sdk/client-s3') as AwsS3Sdk + } catch (e) { + throw new DurableBackendError( + `durable: chdb/durable/s3 needs @aws-sdk/client-s3, which is an optional peer ` + + `dependency so that callers using only the local backend do not pay for it. ` + + `Install it: npm install @aws-sdk/client-s3`, + { cause: e }, + ) + } + return sdk +} + /** Ceiling for a single PutObject. Beyond this a checkpoint needs multipart. */ export const MAX_SINGLE_PUT_BYTES = 5 * 1024 * 1024 * 1024 @@ -92,7 +126,7 @@ export interface S3BackendOptions { /** Key prefix for this object, without a leading slash. May be empty. */ prefix?: string /** Pre-built client. Takes precedence over `clientConfig`. */ - client?: S3Client + client?: S3ClientCtor /** Passed to `new S3Client`. Credentials come from the default chain unless set. */ clientConfig?: S3ClientConfig } @@ -177,7 +211,7 @@ function backendError(what: string, describe: string, e: unknown): DurableBacken export class S3DurableBackend implements DurableBackend { readonly describe: string - private readonly client: S3Client + private readonly client: S3ClientCtor private readonly bucket: string private readonly prefix: string @@ -185,7 +219,7 @@ export class S3DurableBackend implements DurableBackend { if (!options.bucket) throw new RangeError('durable: S3 backend requires a bucket') this.bucket = options.bucket this.prefix = options.prefix ? options.prefix.replace(/^\/+|\/+$/g, '') : '' - this.client = options.client ?? new S3Client(options.clientConfig ?? {}) + this.client = options.client ?? new (awsSdk().S3Client)(options.clientConfig ?? {}) // Never the endpoint's credentials, and never a presigned anything: this // string ends up in error messages and logs. this.describe = `s3://${this.bucket}/${this.prefix}` @@ -206,7 +240,7 @@ export class S3DurableBackend implements DurableBackend { async getBytesWithEtag(key: string): Promise { try { const out = await this.client.send( - new GetObjectCommand({ Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().GetObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key) }), ) if (!out.Body) return undefined const bytes = await out.Body.transformToByteArray() @@ -230,7 +264,7 @@ export class S3DurableBackend implements DurableBackend { async openReadStream(key: string): Promise { try { const out = await this.client.send( - new GetObjectCommand({ Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().GetObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key) }), ) if (!out.Body) return undefined // Under Node the SDK hands back a Readable; the union also covers the @@ -268,7 +302,7 @@ export class S3DurableBackend implements DurableBackend { this.assertWithinSinglePut(key, bytes.byteLength) try { const out = await this.client.send( - new PutObjectCommand({ + new (awsSdk().PutObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key), Body: bytes, @@ -291,11 +325,11 @@ export class S3DurableBackend implements DurableBackend { private async conditionalPut( key: string, - input: Omit[0], 'Bucket' | 'Key'>, + input: Omit[0], 'Bucket' | 'Key'>, ): Promise { try { await this.client.send( - new PutObjectCommand({ ...input, Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().PutObjectCommand)({ ...input, Bucket: this.bucket, Key: this.keyFor(key) }), ) return 'created' } catch (e) { diff --git a/src/durable/namespace.ts b/src/durable/namespace.ts index 250e853..9d00f99 100644 --- a/src/durable/namespace.ts +++ b/src/durable/namespace.ts @@ -57,13 +57,57 @@ export function registerBackendScheme(scheme: string, factory: BackendSchemeFact SCHEMES.set(scheme.replace(/:$/, ''), factory) } -registerBackendScheme('file', (url, objectId) => { - const root = fileURLToPath(url) +/** + * The local filesystem, under two scheme names. + * + * `file:` is this binding's spelling; `local:` is Python's, and Go registers + * the same backend under both for the same reason it matters here — a + * namespace URL is the sort of thing that ends up in a config file shared + * between a Python service and a Node one, and refusing the other binding's + * spelling turns that into a support ticket. Node was the only one of the + * three that took just one. + * + * The two forms differ in how the path is written, not in where it points. + * `file:` is a real URL, so a space is `%20`; `local:` is opaque, so the path + * is closer to literal. Beyond ordinary paths the bindings do not agree + * anyway (Python leaves `local:` undecoded, Go decodes it), so this follows + * Go — one factory, decode both — rather than inventing a third behaviour for + * a case none of them share. + */ +const localFilesystem: BackendSchemeFactory = (url, objectId) => { + const root = url.protocol === 'file:' ? fileURLToPath(url) : opaquePath(url) if (!isAbsolute(root)) { - throw new RangeError(`durable: file namespace must be an absolute path, got ${url.href}`) + throw new RangeError( + `durable: a ${url.protocol} namespace must be an absolute path, got ${url.href}`, + ) } return new LocalDurableBackend({ root: join(root, objectId) }) -}) +} + +/** The path inside a `local:` URL, which is not a `file:` URL and has no host. */ +function opaquePath(url: URL): string { + // A host would be silently dropped otherwise, and `local://data/objects` far + // more likely means a mistyped path than a hostname nobody can resolve. + if (url.host) { + throw new RangeError( + `durable: a ${url.protocol} namespace takes a path, not a host; got ${url.href}. ` + + `Write local:/absolute/path`, + ) + } + try { + return decodeURIComponent(url.pathname) + } catch { + // A stray `%` that is not an escape. Reported here rather than left as a + // bare URIError, which says nothing about which input produced it. + throw new RangeError( + `durable: ${url.href} has a malformed percent-escape in its path; write a literal ` + + `'%' as '%25', or use a file: URL`, + ) + } +} + +registerBackendScheme('file', localFilesystem) +registerBackendScheme('local', localFilesystem) export interface DurableNamespaceOptions { /** How to build the engine for an object being opened. Required. */ diff --git a/test/durable/protocol.test.ts b/test/durable/protocol.test.ts index 52b5a38..c1315ac 100644 --- a/test/durable/protocol.test.ts +++ b/test/durable/protocol.test.ts @@ -13,6 +13,7 @@ import { existsSync } from 'fs' import { chmod, mkdir, mkdtemp, readFile, writeFile } from 'fs/promises' import { tmpdir } from 'os' import { join } from 'path' +import { pathToFileURL } from 'url' import { coldHead, parseHead, serializeHead } from '../../src/durable/head' import { assertEngineCompatible, assertReadable, assertWritable } from '../../src/durable/negotiate' @@ -21,6 +22,8 @@ import { decodeWalSegment, encodeWalSegment, walLineBytes } from '../../src/dura import { checkpointKey, isValidObjectKey, walKey } from '../../src/durable/keys' import { LIMITS } from '../../src/durable/types' import { LocalDurableBackend } from '../../src/durable/backends/local' +import { DurableNamespace } from '../../src/durable/namespace' +import { FakeEngine } from './fakes' import { QueryClass, assertExecuteAllowed, @@ -770,3 +773,54 @@ describe('local backend conditional operations', () => { expect(await be.openReadStream('checkpoints/nope.tar.gz')).toBeUndefined() }) }) + +describe('namespace URL schemes', () => { + // A namespace URL is the sort of thing that ends up in a config file shared + // between a Python service and a Node one. Python spells the local backend + // `local:`, Go registers both, and this binding used to take only `file:` — + // so one config worked in two bindings out of three. + const engineFactory = () => new FakeEngine() + + /** Open and return the failure, which is how the resolved path surfaces. */ + async function openMissing(url: string): Promise { + const ns = new DurableNamespace(url, { engineFactory }) + try { + await ns.open('missing', { readOnly: true }) + throw new Error(`expected ${url} to report a missing object`) + } catch (e) { + return e as Error + } + } + + it('accepts both spellings, and resolves them to the same place', async () => { + const root = await mkdtemp(join(tmpdir(), 'scheme-')) + const dir = join(root, 'ns') + // not_found rather than a scheme error means the backend resolved and + // looked: the object really is absent. + const viaFile = await openMissing(pathToFileURL(dir).href) + const viaLocal = await openMissing(`local:${dir}`) + expect(isDurableErrorOf(viaFile, 'not_found')).toBe(true) + expect(isDurableErrorOf(viaLocal, 'not_found')).toBe(true) + // The message names the resolved location, so equal messages mean both + // spellings point at one directory — which is the whole claim. + expect(viaLocal.message).toBe(viaFile.message) + }) + + it('takes a path containing a space through the local spelling', async () => { + // `file:` percent-encodes a space; `local:` carries it literally, and a + // decoded path is what the other bindings resolve to. + const root = await mkdtemp(join(tmpdir(), 'scheme with space-')) + expect(isDurableErrorOf(await openMissing(`local:${join(root, 'ns')}`), 'not_found')).toBe(true) + }) + + it('refuses a local URL naming a host instead of a path', async () => { + // Dropping the host silently would resolve `local://data/objects` to + // `/objects`, which is not what anyone wrote. + expect((await openMissing('local://data/objects')).message).toMatch(/takes a path, not a host/) + }) + + it('refuses a relative path and a malformed percent-escape', async () => { + expect((await openMissing('local:relative/path')).message).toMatch(/absolute path/) + expect((await openMissing('local:/tmp/100%pct')).message).toMatch(/percent-escape/) + }) +}) diff --git a/test/durable/subpath.test.ts b/test/durable/subpath.test.ts index 96031e5..904815d 100644 --- a/test/durable/subpath.test.ts +++ b/test/durable/subpath.test.ts @@ -131,6 +131,42 @@ describe.skipIf(!built)('subpath imports', () => { expect(out).toContain("chdb/durable/s3") }) + it('loads chdb/durable/s3 even without the AWS SDK, and says what to install', () => { + // The SDK is an optional peer dependency, so a caller using only the local + // backend does not pay for it. That makes "I do want S3 and forgot to + // install it" a real path, and a static import made it a bare + // `Cannot find module '@aws-sdk/client-s3'` from a file the caller has + // never heard of. Checked in a child process with the module resolution + // broken, since the SDK is installed in this tree. + const out = runNode(` + const Module = require('module') + const real = Module._resolveFilename + Module._resolveFilename = function (request, ...rest) { + if (request === '@aws-sdk/client-s3') { + const e = new Error("Cannot find module '@aws-sdk/client-s3'") + e.code = 'MODULE_NOT_FOUND' + throw e + } + return real.call(this, request, ...rest) + } + // The module itself still loads, and still registers the scheme. + const s3 = require('./dist/durable/backends/s3.js') + if (typeof s3.S3DurableBackend !== 'function') throw new Error('no S3DurableBackend') + const { DurableNamespace } = require('./dist/durable/index.js') + new DurableNamespace('s3://b/p?region=us-east-1', { engineFactory: () => ({}) }) + // Only constructing a backend reaches the SDK. + try { + new s3.S3DurableBackend({ bucket: 'b', prefix: 'p' }) + throw new Error('expected a missing-dependency failure') + } catch (e) { + if (!e.message.includes('@aws-sdk/client-s3')) throw new Error('unhelpful: ' + e.message) + if (!e.message.includes('npm install')) throw new Error('no fix given: ' + e.message) + console.log('clean') + } + `) + expect(out).toBe('clean') + }) + it('declares every durable subpath in package.json exports', () => { const pkg = require('../../package.json') as { exports: Record } expect(pkg.exports['./durable']).toBeDefined() From d42eef2af7445ff05e744772adcc016ce617d37e Mon Sep 17 00:00:00 2001 From: Shawn Chen Date: Wed, 16 Sep 2026 10:18:02 +1200 Subject: [PATCH 2/3] Pair the cross-binding check against Go as well MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `chdb-go/chdb/durable` reached V1 too, so the contract's read-by-two-other-bindings requirement was one pair short rather than two. `test:durable:cross-binding` now runs the same four exchanges against Go that it runs against Python: Go writes a base plus a WAL and Node restores it, Node appends and Go reads the append back, and Node checkpoints so Go RESTOREs from that archive with no WAL to fall back on. Both sides load the same engine build — Go finds libchdb through `CHDB_LIB_PATH`, which the script points at this repository's copy — so a difference in results would be a difference in bindings rather than in versions. Worth pinning down: the chdb-go checkout I ran against declared `v26.7.3` in `update_libchdb.sh` while the library on disk was still `26.7.2-rc.2`, which would otherwise have quietly made this a cross-version test wearing a cross-binding label. A pairing whose runtime is absent is skipped loudly and the rest still runs; only having neither fails. Go needs its `ScratchRoot` to exist, same as Node. Co-Authored-By: Claude Opus 5 (1M context) --- docs/design/durable-control-plane.md | 48 ++++--- scripts/durable-cross-binding.mjs | 204 +++++++++++++++++++++++++-- 2 files changed, 220 insertions(+), 32 deletions(-) diff --git a/docs/design/durable-control-plane.md b/docs/design/durable-control-plane.md index dc886a9..89c74cb 100644 --- a/docs/design/durable-control-plane.md +++ b/docs/design/durable-control-plane.md @@ -437,10 +437,17 @@ them at adoption time rather than per-commit is low: CHDB_LIBCHDB_OLD=…/26.7.2-rc.2/libchdb.so CHDB_LIBCHDB_NEW=…/26.7.3/libchdb.so \ npm run test:durable:cross-version # a newer engine restores an older one's archive -CHDB_PYTHON=/tmp/pyenv/bin/python \ - npm run test:durable:cross-binding # Python writes, Node reads, and back +CHDB_PYTHON=/tmp/pyenv/bin/python CHDB_GO_REPO=/path/to/chdb-go \ + npm run test:durable:cross-binding # Python and Go write, Node reads, and back ``` +`cross-binding` skips a pairing whose runtime is absent and fails only when +neither is there, so it is worth running with just one installed. Go finds its +engine through `CHDB_LIB_PATH`, which the script points at this repository's +`libchdb.so` — worth knowing because the chdb-go checkout I ran against +declared `v26.7.3` while the library on disk was still `26.7.2-rc.2`, which +would have made this a cross-version test wearing a cross-binding label. + `cross-version` re-execs itself once per stage. A libchdb is `dlopen`ed once per process — it has to be, since the engine binds one data path per process — so a single-process version would load the first library and answer every @@ -531,7 +538,7 @@ Against the V1 conformance list, this binding's position: | Restoring full backups from *earlier* core releases | covered — `npm run test:durable:cross-version` | | Old-header/new-library ABI | covered — an addon compiled against `26.7.2-rc.2` headers runs the whole suite on `26.7.3` | | New-header/old-library ABI | not covered — needs an addon built against headers newer than the engine it loads, which no released pair produces yet | -| Every writer's fixture read by two other bindings | Python covered — `npm run test:durable:cross-binding`; Go pending | +| Every writer's fixture read by two other bindings | covered — Python and Go, `npm run test:durable:cross-binding` | Two of these became testable when `26.7.3` shipped, since `26.7.2-rc.2` then stopped being the only Durable-capable engine. Both scripts are run by hand @@ -546,19 +553,22 @@ Measured on `26.7.2-rc.2` → `26.7.3`: | …and replays a WAL segment the old engine appended | holds | | Writing raises `min_reader`, and the old engine is then refused `engine_incompatible` | holds | -Measured against Python `chdb` 4.4.0 (same `26.7.3` engine, its own bindings): +Measured against Python `chdb` 4.4.0 and `chdb-go` (all three on the same +`26.7.3` engine, each through its own bindings, so a difference would be a +binding difference rather than a version one): -| Exchange | Result | -| --- | --- | -| Python base + WAL, restored and replayed by Node | holds | -| Node appends to a Python-written object; Python reads it back | holds | -| Node checkpoints; Python RESTOREs from that archive with no WAL left | holds | -| Both on a real object store (MinIO), not just a local directory | holds | -| One `local:` URL string opened by both | holds | +| Exchange | Python | Go | +| --- | --- | --- | +| Its base + WAL, restored and replayed by Node | holds | holds | +| Node appends to its object; it reads the append back | holds | holds | +| Node checkpoints; it RESTOREs from that archive with no WAL left | holds | holds | +| One `local:` URL string opened by both | holds | — | +| Also over a real object store (MinIO), not just a local directory | holds | — | -Go reached V1 as well (`chdb-go/chdb/durable`), so the contract's -read-by-two-other-bindings requirement is one pair short rather than two. The -Go exchange has not been run. +That covers the contract's requirement from this binding's side: an object Node +wrote is read by both other V1 implementations, and objects they wrote are read +by Node. The pairing the three still owe each other is Python ↔ Go, which is +not this repository's to run. ### Provider conformance @@ -720,9 +730,7 @@ this package: which the loader prefers over a local build, and *that* is the case the adapter refuses by name. Until a tag goes out, using `chdb/durable/node` from a checkout means `npm run build` plus `rm -rf node_modules/@chdb/lib-*`. -2. Run the cross-binding exchange against Go as well. Python is covered by - `npm run test:durable:cross-binding`, and `chdb-go/chdb/durable` reached V1 - too, so the contract's read-by-two-other-bindings requirement is one pair - short. The shared fixtures in the chdb repository are still worth adopting - when they exist — a fixture pins a byte layout, while these scripts only - prove two live implementations agree. +2. Adopt the shared cross-binding fixtures from the chdb repository once they + exist. `test:durable:cross-binding` proves Node, Python and Go agree as + live implementations, which is not quite the same claim — a fixture pins a + byte layout, so it catches three implementations drifting together. diff --git a/scripts/durable-cross-binding.mjs b/scripts/durable-cross-binding.mjs index 35aaf0c..27e6a0d 100644 --- a/scripts/durable-cross-binding.mjs +++ b/scripts/durable-cross-binding.mjs @@ -4,8 +4,8 @@ * * The V1 contract's cross-binding requirement is that a writer's object be * readable by the other implementations — the on-disk format is the interface, - * not any one library's API. Python (`chdb` ≥ 4.4.0) and Go both reached V1, - * so this pairs Node against Python; Go is not covered here. + * not any one library's API. Python (`chdb` >= 4.4.0) and Go have both reached + * V1, and this pairs Node against each of them. * * Four exchanges, because each proves something the others do not: * @@ -19,19 +19,30 @@ * 4. Both spellings of the local backend URL name the same object — Python * writes `local:`, Node reads the identical string. * + * Each pairing is skipped, loudly, when its runtime is absent — so the script + * is useful with either one installed. + * * ```sh * python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0' + * git clone https://github.com/chdb-io/chdb-go /tmp/chdb-go * npm run build # the addon must carry the durable ABI - * CHDB_PYTHON=/tmp/pyenv/bin/python node scripts/durable-cross-binding.mjs + * + * CHDB_PYTHON=/tmp/pyenv/bin/python \ + * CHDB_GO_REPO=/tmp/chdb-go \ + * node scripts/durable-cross-binding.mjs * ``` * - * Not in CI: it needs a second language runtime with its own engine build. + * Go loads its engine through `CHDB_LIB_PATH`, which this script points at + * this repository's `libchdb.so` — so both sides run the same engine build + * and a difference in results is a difference in bindings, not in versions. + * + * Not in CI: it needs other language runtimes with their own engine builds. * Run it when adopting an engine, and record the result in * docs/design/durable-control-plane.md. */ import { execFileSync } from 'child_process' -import { mkdtempSync, writeFileSync, rmSync } from 'fs' +import { mkdirSync, mkdtempSync, writeFileSync, rmSync } from 'fs' import { tmpdir } from 'os' import { join } from 'path' import { createRequire } from 'module' @@ -52,16 +63,107 @@ function pythonHasDurable() { } const pyVersion = pythonHasDurable() -if (!pyVersion) { - console.error( - `${PYTHON} has no chdb.durable. Install it:\n` + - ` python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0'\n` + - ` CHDB_PYTHON=/tmp/pyenv/bin/python node scripts/durable-cross-binding.mjs`, + +const { DurableNamespace, nodeEngineFactory } = require('./dist/durable/node.js') + +const GO_REPO = process.env.CHDB_GO_REPO +const LIBCHDB = join(process.cwd(), 'libchdb.so') + +function goAvailable() { + try { + execFileSync('go', ['version'], { stdio: 'ignore' }) + return true + } catch { + return false + } +} + +/** + * The Go side, as a throwaway module that `replace`s chdb-go to the checkout. + * + * Built rather than `go run` so a compile error surfaces once, before any + * object is written, instead of once per invocation. + */ +function buildGoHelper() { + const dir = mkdtempSync(join(tmpdir(), 'xbind-go-src-')) + writeFileSync( + join(dir, 'go.mod'), + `module xbindgo\n\ngo 1.24\n\nrequire github.com/chdb-io/chdb-go/v2 v2.0.0\n\n` + + `replace github.com/chdb-io/chdb-go/v2 => ${GO_REPO}\n`, ) - process.exit(2) + writeFileSync(join(dir, 'main.go'), GO_MAIN) + // The checkout's own go.sum covers these; tidy writes ours from it. + execFileSync('go', ['mod', 'tidy'], { cwd: dir, stdio: 'ignore', env: goEnv() }) + const bin = join(dir, 'xbindgo') + execFileSync('go', ['build', '-o', bin, '.'], { cwd: dir, stdio: 'inherit', env: goEnv() }) + return bin } -const { DurableNamespace, nodeEngineFactory } = require('./dist/durable/node.js') +/** Go finds its engine through CHDB_LIB_PATH; point it at ours so both sides + * run one engine build and a difference is a binding difference. */ +function goEnv() { + return { ...process.env, CHDB_LIB_PATH: LIBCHDB } +} + +function runGo(bin, args) { + return execFileSync(bin, args, { encoding: 'utf8', env: goEnv() }) +} + +const GO_MAIN = `package main + +import ( + "context" + "fmt" + "os" + + "github.com/chdb-io/chdb-go/v2/chdb/durable" +) + +func main() { + phase, root := os.Args[1], os.Args[2] + ctx := context.Background() + + ns, err := durable.NewNamespace("file://"+root+"/ns", durable.NamespaceOptions{ + ScratchRoot: root + "/scratch-go", + Tuning: durable.Tuning{LeaseTTL: 120e9, HeartbeatInterval: 30e9}, + }) + must(err) + + switch phase { + case "write": + obj, _, err := ns.Open(ctx, "goobj", durable.OpenOptions{Database: "default"}) + must(err) + _, err = obj.Execute(ctx, "CREATE TABLE g (id UInt64, note String) ENGINE = MergeTree ORDER BY id") + must(err) + _, err = obj.Execute(ctx, "INSERT INTO g VALUES (1, 'go-one'), (2, 'go-two')") + must(err) + _, err = obj.Checkpoint(ctx) + must(err) + _, err = obj.Execute(ctx, "INSERT INTO g VALUES (3, 'go-three')") + must(err) + _, err = obj.Flush(ctx) + must(err) + must(obj.Close(ctx)) + case "read": + obj, _, err := ns.Open(ctx, os.Args[3], durable.OpenOptions{}) + must(err) + out, err := obj.Query(ctx, "SELECT id, note FROM "+os.Args[4]+" ORDER BY id", "CSV") + must(err) + fmt.Print(out) + must(obj.Close(ctx)) + default: + fmt.Fprintln(os.Stderr, "unknown phase "+phase) + os.Exit(2) + } +} + +func must(err error) { + if err != nil { + fmt.Fprintln(os.Stderr, "ERR:", err) + os.Exit(1) + } +} +` const root = mkdtempSync(join(tmpdir(), 'durable-xbind-')) let failed = false @@ -93,7 +195,17 @@ const NODE_ENGINE = await (async () => { })() console.log(`node addon engine ${NODE_ENGINE} <-> python chdb ${pyVersion}\n`) +let ran = 0 + +if (!pyVersion) { + console.log( + `SKIP Python — ${PYTHON} has no chdb.durable.\n` + + ` python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0'\n`, + ) +} + try { + if (pyVersion) { // 1. Python writes base + WAL. python(` import os @@ -185,6 +297,74 @@ o.close() report('one local: URL serves both bindings', rows === '"python"\n', rows.trim()) await obj.close() } + ran++ + } + + // ---- Go ---- + if (!GO_REPO) { + console.log( + '\nSKIP Go — set CHDB_GO_REPO to a chdb-go checkout.\n' + + ' git clone https://github.com/chdb-io/chdb-go /tmp/chdb-go\n', + ) + } else if (!goAvailable()) { + console.log('\nSKIP Go — no `go` on PATH.\n') + } else { + console.log('') + const bin = buildGoHelper() + // Go does not create its ScratchRoot either. + mkdirSync(join(root, 'scratch-go'), { recursive: true }) + + // Go writes a base plus a WAL segment on top. + runGo(bin, ['write', root]) + + // Node restores the Go base and replays the Go WAL, then appends. + { + const ns = namespace(`file://${root}/ns`, 'xbind-go') + const phases = [] + const obj = await ns.open('goobj', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM g ORDER BY id', { format: 'CSV' }) + report('Node restores a Go base', phases.includes('restoring-base')) + report('Node replays a Go WAL', phases.includes('replaying-wal')) + report( + 'Go -> Node rows match', + rows === '1,"go-one"\n2,"go-two"\n3,"go-three"\n', + rows.trim().replace(/\n/g, ' | '), + ) + await obj.execute("INSERT INTO g VALUES (4, 'node-four')") + await obj.flush() + await obj.close() + } + + // Go reads Node's append back. + { + const out = runGo(bin, ['read', root, 'goobj', 'g']) + report( + 'Node -> Go rows match', + out === '1,"go-one"\n2,"go-two"\n3,"go-three"\n4,"node-four"\n', + out.trim().replace(/\n/g, ' | '), + ) + } + + // Node checkpoints; Go restores from that archive with no WAL left. + { + const ns = namespace(`file://${root}/ns`, 'xbind-go-nw') + const obj = await ns.open('nodeobj-go', { database: 'default' }) + await obj.execute('CREATE TABLE n (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO n VALUES (1, 'from-node')") + await obj.checkpoint() + report('Node checkpoint left no WAL (for Go)', obj.manifest.wal.length === 0) + await obj.close() + + const out = runGo(bin, ['read', root, 'nodeobj-go', 'n']) + report("Go RESTOREs Node's archive", out === '1,"from-node"\n', out.trim()) + } + ran++ + } + + if (ran === 0) { + console.error('\nNo other binding was available, so nothing was verified.') + failed = true + } } finally { rmSync(root, { recursive: true, force: true }) } From a2afdfbb791cfe3f096a77e4ab62c4601e4a433a Mon Sep 17 00:00:00 2001 From: Shawn Chen Date: Thu, 17 Sep 2026 16:51:46 +1200 Subject: [PATCH 3/3] Give the 12 MiB case a timeout a real bucket can meet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The large-object test moves 12 MiB up and back down. Against a local MinIO that is 0.4s; against a real bucket across an ocean it is about a minute, and the suite's 20s budget made it the one test that fails on AWS while the implementation is fine — measured, at 20005ms, on `eu-central-1`. With its own 180s it passes there and stays unchanged everywhere else. The suite-wide 20s is left alone on purpose: it is a useful constraint on the other ten cases, none of which moves more than a few hundred bytes. A red suite that means "your network is far away" is one people learn to ignore, which is worse than a slow test. Co-Authored-By: Claude Opus 5 (1M context) --- docs/design/durable-control-plane.md | 9 ++++++++- test/durable/s3-backend.test.ts | 7 ++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/design/durable-control-plane.md b/docs/design/durable-control-plane.md index 89c74cb..710477d 100644 --- a/docs/design/durable-control-plane.md +++ b/docs/design/durable-control-plane.md @@ -604,9 +604,16 @@ Run against engine `26.7.3`: | Provider | Status | | --- | --- | | MinIO (latest) | 11/11 conformance, plus 4/4 full stack, plus the cross-binding exchange | -| AWS S3 (`us-east-2`) | 11/11 conformance and 4/4 full stack on `26.7.2-rc.2`; not re-run on `26.7.3` | +| AWS S3 (`eu-central-1`) | 11/11 conformance, 4/4 full stack, and the cross-binding exchange, on `26.7.3` | | Cloudflare R2 | not run — the code path is the same, but the claim is not made until measured | +One number is worth carrying: the 12 MiB large-object case takes about 0.4s +against a local MinIO and about a minute against a real bucket an ocean away, +so it carries its own 180s timeout rather than the suite's 20s. Without that +it is the one test that fails on AWS while the implementation is fine, and a +red suite meaning "your network is far away" is a suite people learn to +ignore. + The two that have run agree on every point, including the ETag behaviour above: `If-None-Match: *` and `If-Match` are atomic on both, an eight-way race leaves exactly one winner on both, and re-PUTting identical bytes advances the token on diff --git a/test/durable/s3-backend.test.ts b/test/durable/s3-backend.test.ts index c0b9523..ef54d7f 100644 --- a/test/durable/s3-backend.test.ts +++ b/test/durable/s3-backend.test.ts @@ -179,6 +179,11 @@ describe.skipIf(!BUCKET)(`S3-compatible backend (${ENDPOINT ?? 'aws'})`, () => { }) describe('large objects', () => { + // 180s rather than the suite's 20s. 12 MiB is nothing against a local + // MinIO — 0.4s — and about a minute against a real bucket across an + // ocean, so the default made this the one test that fails on AWS while + // the implementation is fine. A red suite that means "your network is + // far away" trains people to ignore it. it('streams a file up and back down with its digest intact', async () => { const be = backend('stream') const dir = await mkdtemp(join(tmpdir(), 'durable-s3-')) @@ -206,7 +211,7 @@ describe.skipIf(!BUCKET)(`S3-compatible backend (${ENDPOINT ?? 'aws'})`, () => { }) expect(size).toBe(expected.size) expect(hash.digest('hex')).toBe(expected.sha256) - }) + }, 180_000) }) describe('recovery on another machine', () => {