diff --git a/README.md b/README.md index 024ce0b..98358e3 100644 --- a/README.md +++ b/README.md @@ -226,6 +226,10 @@ object needs a private one, so a process holds one open durable object at a time and no ordinary `Session` beside it — fan-out goes across worker processes. +The local backend answers to both `file:` and `local:`, because a namespace URL +gets shared between services in different languages and Python spells it +`local:` while Go takes either. + Recovery on another machine needs the object to live somewhere neither machine owns, so `chdb/durable/s3` provides an S3-compatible backend — AWS S3, Cloudflare R2 and MinIO through one implementation. It sits behind its own diff --git a/docs/design/durable-control-plane.md b/docs/design/durable-control-plane.md index 84e7119..710477d 100644 --- a/docs/design/durable-control-plane.md +++ b/docs/design/durable-control-plane.md @@ -266,6 +266,19 @@ can plant it can rewrite the objects directly — and check-then-use cannot be made atomic without `openat`, which Node does not expose. Anything wanting that property should not be on this backend. +It answers to two scheme names, `file:` and `local:`. That is not a +convenience — a namespace URL ends up in a config file, and that config file +gets shared between services in different languages. Python spells this +backend `local:`, Go registers both, and this binding took only `file:` until +it was pointed out, which made Node the one place a working configuration +stopped working. The two forms differ in how a path is written rather than +where it points: `file:` is a real URL, so a space is `%20`; `local:` is +opaque, so the path is closer to literal. Past ordinary paths the bindings do +not agree anyway — Python leaves `local:` undecoded, Go decodes it — so this +follows Go rather than inventing a third behaviour for a case none of them +share. A `local:` URL naming a host (`local://data/objects`) is refused +instead of silently resolving to `/objects`. + A local directory cannot be a remote authority. When the machine holding it is gone, so is the object — so the local backend is what conformance and development run on, not what makes a database recoverable somewhere else. @@ -416,6 +429,31 @@ npm run test:durable:e2e # end-to-end against a real libchdb, under Bun npm run test:durable:stack # the whole stack: Bun + libchdb + real object storage ``` +Two more run by hand, because each needs something CI does not have. They +check the promises that only move when core releases, so the cost of running +them at adoption time rather than per-commit is low: + +```sh +CHDB_LIBCHDB_OLD=…/26.7.2-rc.2/libchdb.so CHDB_LIBCHDB_NEW=…/26.7.3/libchdb.so \ + npm run test:durable:cross-version # a newer engine restores an older one's archive + +CHDB_PYTHON=/tmp/pyenv/bin/python CHDB_GO_REPO=/path/to/chdb-go \ + npm run test:durable:cross-binding # Python and Go write, Node reads, and back +``` + +`cross-binding` skips a pairing whose runtime is absent and fails only when +neither is there, so it is worth running with just one installed. Go finds its +engine through `CHDB_LIB_PATH`, which the script points at this repository's +`libchdb.so` — worth knowing because the chdb-go checkout I ran against +declared `v26.7.3` while the library on disk was still `26.7.2-rc.2`, which +would have made this a cross-version test wearing a cross-binding label. + +`cross-version` re-execs itself once per stage. A libchdb is `dlopen`ed once +per process — it has to be, since the engine binds one data path per process — +so a single-process version would load the first library and answer every +later request with it: both versions report the same number, the floor never +rises, and the script passes while proving nothing. + The last one is the arrangement a downstream actually deploys, and the only place the other two suites' halves meet. Each of them covers one: a real engine over a local directory, or a real bucket under a fake engine. Both can pass @@ -453,7 +491,7 @@ CHDB_LIBCHDB_PATH=/path/to/chdb-core/buildlib/libchdb.so npm run test:durable:e2 `CHDB_LIBCHDB_PATH` is only needed when a platform package that predates the durable ABI is installed. The resolver prefers `@chdb/lib-*` over anything in the working tree, so a stale one shadows a local `chdb-core` build — and until -the RC platform packages are published, the newest on npm is +a Durable-capable platform package is published, the newest on npm is `26.7.0-stable.1`, which exports none of the durable symbols. Either override the path or `rm -rf node_modules/@chdb/lib-*`, which is what CI already does. Running against a shadowing build fails with a message naming the missing ABI @@ -497,12 +535,40 @@ Against the V1 conformance list, this binding's position: | Single-writer races, fencing, heartbeat, failed and ambiguous commits | covered | | Unknown-field round trips, size limits, secret redaction, close failure | covered | | Full provider suite against a real object store | covered — AWS S3 and MinIO | -| Restoring full backups from *earlier* core releases | not applicable yet — `v26.7.2-rc.2` is the first release with the Durable ABI, so there is no earlier archive to restore | -| New-header/old-library and old-header/new-library ABI tests | not applicable yet — same reason; the promise starts here | -| Every writer's fixture read by two other bindings | pending — needs Python and Go to reach V1 | +| Restoring full backups from *earlier* core releases | covered — `npm run test:durable:cross-version` | +| Old-header/new-library ABI | covered — an addon compiled against `26.7.2-rc.2` headers runs the whole suite on `26.7.3` | +| New-header/old-library ABI | not covered — needs an addon built against headers newer than the engine it loads, which no released pair produces yet | +| Every writer's fixture read by two other bindings | covered — Python and Go, `npm run test:durable:cross-binding` | -The two "not applicable" rows become real on the second Durable-capable core -release, and the fixtures written now are what they will be tested against. +Two of these became testable when `26.7.3` shipped, since `26.7.2-rc.2` then +stopped being the only Durable-capable engine. Both scripts are run by hand +rather than in CI — one needs two engine downloads, the other a second +language runtime — and what they check moves only when core releases. + +Measured on `26.7.2-rc.2` → `26.7.3`: + +| Claim | Result | +| --- | --- | +| New engine restores a base archive the old engine wrote | holds | +| …and replays a WAL segment the old engine appended | holds | +| Writing raises `min_reader`, and the old engine is then refused `engine_incompatible` | holds | + +Measured against Python `chdb` 4.4.0 and `chdb-go` (all three on the same +`26.7.3` engine, each through its own bindings, so a difference would be a +binding difference rather than a version one): + +| Exchange | Python | Go | +| --- | --- | --- | +| Its base + WAL, restored and replayed by Node | holds | holds | +| Node appends to its object; it reads the append back | holds | holds | +| Node checkpoints; it RESTOREs from that archive with no WAL left | holds | holds | +| One `local:` URL string opened by both | holds | — | +| Also over a real object store (MinIO), not just a local directory | holds | — | + +That covers the contract's requirement from this binding's side: an object Node +wrote is read by both other V1 implementations, and objects they wrote are read +by Node. The pairing the three still owe each other is Python ↔ Go, which is +not this repository's to run. ### Provider conformance @@ -533,14 +599,21 @@ durable object written on one "machine" — one namespace instance, one engine, one scratch tree — reopened on another that shares nothing but the bucket, for both a WAL-only object and a checkpointed one. -Run against engine `26.7.2-rc.2`: +Run against engine `26.7.3`: | Provider | Status | | --- | --- | -| AWS S3 (`us-east-2`) | 11/11 conformance, plus 4/4 full stack | -| MinIO (`RELEASE.2025-09-07`) | 11/11 conformance, run repeatedly | +| MinIO (latest) | 11/11 conformance, plus 4/4 full stack, plus the cross-binding exchange | +| AWS S3 (`eu-central-1`) | 11/11 conformance, 4/4 full stack, and the cross-binding exchange, on `26.7.3` | | Cloudflare R2 | not run — the code path is the same, but the claim is not made until measured | +One number is worth carrying: the 12 MiB large-object case takes about 0.4s +against a local MinIO and about a minute against a real bucket an ocean away, +so it carries its own 180s timeout rather than the suite's 20s. Without that +it is the one test that fails on AWS while the implementation is fine, and a +red suite meaning "your network is far away" is a suite people learn to +ignore. + The two that have run agree on every point, including the ETag behaviour above: `If-None-Match: *` and `If-Match` are atomic on both, an eight-way race leaves exactly one winner on both, and re-PUTting identical bytes advances the token on @@ -664,5 +737,7 @@ this package: which the loader prefers over a local build, and *that* is the case the adapter refuses by name. Until a tag goes out, using `chdb/durable/node` from a checkout means `npm run build` plus `rm -rf node_modules/@chdb/lib-*`. -2. Add the shared cross-binding fixtures from the chdb repository once they - exist, and read a Python-written object with them. +2. Adopt the shared cross-binding fixtures from the chdb repository once they + exist. `test:durable:cross-binding` proves Node, Python and Go agree as + live implementations, which is not quite the same claim — a fixture pins a + byte layout, so it catches three implementations drifting together. diff --git a/package.json b/package.json index b644d8e..ad849c9 100644 --- a/package.json +++ b/package.json @@ -77,7 +77,9 @@ "test:durable:node": "npm run build:ts && vitest run --config vitest.durable-node.config.ts", "test:durable:e2e": "bun test --timeout 120000 test/durable/libchdb-e2e.bun.test.ts", "test:durable:s3": "vitest run --config vitest.durable.config.ts test/durable/s3-backend.test.ts", - "test:durable:stack": "bun test --timeout 300000 test/durable/full-stack.bun.test.ts" + "test:durable:stack": "bun test --timeout 300000 test/durable/full-stack.bun.test.ts", + "test:durable:cross-version": "bun scripts/durable-cross-version.ts", + "test:durable:cross-binding": "npm run build:ts && node scripts/durable-cross-binding.mjs" }, "author": { "name": "chdb", diff --git a/scripts/durable-cross-binding.mjs b/scripts/durable-cross-binding.mjs new file mode 100644 index 0000000..27e6a0d --- /dev/null +++ b/scripts/durable-cross-binding.mjs @@ -0,0 +1,373 @@ +#!/usr/bin/env node +/** + * Can another binding read what this one wrote, and the other way round? + * + * The V1 contract's cross-binding requirement is that a writer's object be + * readable by the other implementations — the on-disk format is the interface, + * not any one library's API. Python (`chdb` >= 4.4.0) and Go have both reached + * V1, and this pairs Node against each of them. + * + * Four exchanges, because each proves something the others do not: + * + * 1. Python writes a base archive plus a WAL segment; Node opens it. Proves + * Node can RESTORE a foreign archive and replay a foreign WAL. + * 2. Node appends to that same object; Python reads it back. Proves a + * foreign writer's head/CAS/lease round trip is accepted. + * 3. Node creates its own object and checkpoints; Python restores purely + * from Node's base, with no WAL left. The strongest direction: Python's + * engine RESTOREs an archive Node's BACKUP produced. + * 4. Both spellings of the local backend URL name the same object — Python + * writes `local:`, Node reads the identical string. + * + * Each pairing is skipped, loudly, when its runtime is absent — so the script + * is useful with either one installed. + * + * ```sh + * python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0' + * git clone https://github.com/chdb-io/chdb-go /tmp/chdb-go + * npm run build # the addon must carry the durable ABI + * + * CHDB_PYTHON=/tmp/pyenv/bin/python \ + * CHDB_GO_REPO=/tmp/chdb-go \ + * node scripts/durable-cross-binding.mjs + * ``` + * + * Go loads its engine through `CHDB_LIB_PATH`, which this script points at + * this repository's `libchdb.so` — so both sides run the same engine build + * and a difference in results is a difference in bindings, not in versions. + * + * Not in CI: it needs other language runtimes with their own engine builds. + * Run it when adopting an engine, and record the result in + * docs/design/durable-control-plane.md. + */ + +import { execFileSync } from 'child_process' +import { mkdirSync, mkdtempSync, writeFileSync, rmSync } from 'fs' +import { tmpdir } from 'os' +import { join } from 'path' +import { createRequire } from 'module' + +const require = createRequire(join(process.cwd(), 'package.json')) +const PYTHON = process.env.CHDB_PYTHON ?? 'python3' + +function pythonHasDurable() { + try { + const v = execFileSync(PYTHON, ['-c', 'import chdb,chdb.durable;print(chdb.__version__)'], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim() + return v + } catch { + return null + } +} + +const pyVersion = pythonHasDurable() + +const { DurableNamespace, nodeEngineFactory } = require('./dist/durable/node.js') + +const GO_REPO = process.env.CHDB_GO_REPO +const LIBCHDB = join(process.cwd(), 'libchdb.so') + +function goAvailable() { + try { + execFileSync('go', ['version'], { stdio: 'ignore' }) + return true + } catch { + return false + } +} + +/** + * The Go side, as a throwaway module that `replace`s chdb-go to the checkout. + * + * Built rather than `go run` so a compile error surfaces once, before any + * object is written, instead of once per invocation. + */ +function buildGoHelper() { + const dir = mkdtempSync(join(tmpdir(), 'xbind-go-src-')) + writeFileSync( + join(dir, 'go.mod'), + `module xbindgo\n\ngo 1.24\n\nrequire github.com/chdb-io/chdb-go/v2 v2.0.0\n\n` + + `replace github.com/chdb-io/chdb-go/v2 => ${GO_REPO}\n`, + ) + writeFileSync(join(dir, 'main.go'), GO_MAIN) + // The checkout's own go.sum covers these; tidy writes ours from it. + execFileSync('go', ['mod', 'tidy'], { cwd: dir, stdio: 'ignore', env: goEnv() }) + const bin = join(dir, 'xbindgo') + execFileSync('go', ['build', '-o', bin, '.'], { cwd: dir, stdio: 'inherit', env: goEnv() }) + return bin +} + +/** Go finds its engine through CHDB_LIB_PATH; point it at ours so both sides + * run one engine build and a difference is a binding difference. */ +function goEnv() { + return { ...process.env, CHDB_LIB_PATH: LIBCHDB } +} + +function runGo(bin, args) { + return execFileSync(bin, args, { encoding: 'utf8', env: goEnv() }) +} + +const GO_MAIN = `package main + +import ( + "context" + "fmt" + "os" + + "github.com/chdb-io/chdb-go/v2/chdb/durable" +) + +func main() { + phase, root := os.Args[1], os.Args[2] + ctx := context.Background() + + ns, err := durable.NewNamespace("file://"+root+"/ns", durable.NamespaceOptions{ + ScratchRoot: root + "/scratch-go", + Tuning: durable.Tuning{LeaseTTL: 120e9, HeartbeatInterval: 30e9}, + }) + must(err) + + switch phase { + case "write": + obj, _, err := ns.Open(ctx, "goobj", durable.OpenOptions{Database: "default"}) + must(err) + _, err = obj.Execute(ctx, "CREATE TABLE g (id UInt64, note String) ENGINE = MergeTree ORDER BY id") + must(err) + _, err = obj.Execute(ctx, "INSERT INTO g VALUES (1, 'go-one'), (2, 'go-two')") + must(err) + _, err = obj.Checkpoint(ctx) + must(err) + _, err = obj.Execute(ctx, "INSERT INTO g VALUES (3, 'go-three')") + must(err) + _, err = obj.Flush(ctx) + must(err) + must(obj.Close(ctx)) + case "read": + obj, _, err := ns.Open(ctx, os.Args[3], durable.OpenOptions{}) + must(err) + out, err := obj.Query(ctx, "SELECT id, note FROM "+os.Args[4]+" ORDER BY id", "CSV") + must(err) + fmt.Print(out) + must(obj.Close(ctx)) + default: + fmt.Fprintln(os.Stderr, "unknown phase "+phase) + os.Exit(2) + } +} + +func must(err error) { + if err != nil { + fmt.Fprintln(os.Stderr, "ERR:", err) + os.Exit(1) + } +} +` + +const root = mkdtempSync(join(tmpdir(), 'durable-xbind-')) +let failed = false + +function report(claim, ok, detail = '') { + console.log(`${ok ? 'PASS' : 'FAIL'} ${claim}${detail ? ` — ${detail}` : ''}`) + if (!ok) failed = true +} + +/** Run a Python snippet against the shared namespace; returns its stdout. */ +function python(body) { + const file = join(root, `py-${Math.random().toString(36).slice(2)}.py`) + writeFileSync(file, body) + return execFileSync(PYTHON, [file], { encoding: 'utf8', env: { ...process.env, XBIND_ROOT: root } }) +} + +/** A namespace on the shared directory, driven by this package's addon. */ +function namespace(url, scratchName) { + return new DurableNamespace(url, { + engineFactory: nodeEngineFactory(), + scratchRoot: mkdtempSync(join(tmpdir(), `${scratchName}-`)), + tuning: { leaseTtlMs: 120_000, heartbeatIntervalMs: 30_000 }, + }) +} + +const NODE_ENGINE = await (async () => { + const { ChdbNodeEngine } = require('./dist/durable/node.js') + return new ChdbNodeEngine().version() +})() +console.log(`node addon engine ${NODE_ENGINE} <-> python chdb ${pyVersion}\n`) + +let ran = 0 + +if (!pyVersion) { + console.log( + `SKIP Python — ${PYTHON} has no chdb.durable.\n` + + ` python3 -m venv /tmp/pyenv && /tmp/pyenv/bin/pip install 'chdb>=4.4.0'\n`, + ) +} + +try { + if (pyVersion) { + // 1. Python writes base + WAL. + python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('shared') +o.execute("CREATE TABLE events (id UInt64, note String) ENGINE = MergeTree ORDER BY id") +o.execute("INSERT INTO events VALUES (1, 'py-one'), (2, 'py-two')") +o.checkpoint() +o.execute("INSERT INTO events VALUES (3, 'py-three')") +o.flush() +o.close() +`) + + // 2. Node restores it, then appends. + { + const ns = namespace(`file://${root}/ns`, 'xbind-node') + const phases = [] + const obj = await ns.open('shared', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM events ORDER BY id', { format: 'CSV' }) + report('Node restores a Python base', phases.includes('restoring-base')) + report('Node replays a Python WAL', phases.includes('replaying-wal')) + report( + 'Python -> Node rows match', + rows === '1,"py-one"\n2,"py-two"\n3,"py-three"\n', + rows.trim().replace(/\n/g, ' | '), + ) + await obj.execute("INSERT INTO events VALUES (4, 'node-four')") + await obj.flush() + await obj.close() + } + + // 3. Python reads Node's append. + { + const out = python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('shared') +print(o.query('SELECT id, note FROM events ORDER BY id','CSV').bytes().decode(), end='') +o.close() +`) + report( + 'Node -> Python rows match', + out === '1,"py-one"\n2,"py-two"\n3,"py-three"\n4,"node-four"\n', + out.trim().replace(/\n/g, ' | '), + ) + } + + // 4. Node creates its own object and checkpoints; Python restores from that + // base alone, with no WAL to fall back on. + { + const ns = namespace(`file://${root}/ns`, 'xbind-nw') + const obj = await ns.open('nodeobj', { database: 'default' }) + await obj.execute('CREATE TABLE t (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO t VALUES (1, 'from-node')") + await obj.checkpoint() + report('Node checkpoint left no WAL', obj.manifest.wal.length === 0) + await obj.close() + + const out = python(` +import os +from chdb.durable import Namespace +ns = Namespace('file://' + os.environ['XBIND_ROOT'] + '/ns', db='default') +o = ns.open('nodeobj') +assert len(o.wal) == 0, 'expected a checkpoint-only object' +print(o.query('SELECT id, note FROM t','CSV').bytes().decode(), end='') +o.close() +`) + report("Python RESTOREs Node's archive", out === '1,"from-node"\n', out.trim()) + } + + // 5. One `local:` URL string, both bindings. + { + const url = `local:${root}/shared-url` + python(` +import os +from chdb.durable import Namespace +ns = Namespace('local:' + os.environ['XBIND_ROOT'] + '/shared-url', db='default') +o = ns.open('o') +o.execute("CREATE TABLE u (who String) ENGINE = MergeTree ORDER BY who") +o.execute("INSERT INTO u VALUES ('python')") +o.flush() +o.close() +`) + const ns = namespace(url, 'xbind-url') + const obj = await ns.open('o') + const rows = await obj.query('SELECT who FROM u', { format: 'CSV' }) + report('one local: URL serves both bindings', rows === '"python"\n', rows.trim()) + await obj.close() + } + ran++ + } + + // ---- Go ---- + if (!GO_REPO) { + console.log( + '\nSKIP Go — set CHDB_GO_REPO to a chdb-go checkout.\n' + + ' git clone https://github.com/chdb-io/chdb-go /tmp/chdb-go\n', + ) + } else if (!goAvailable()) { + console.log('\nSKIP Go — no `go` on PATH.\n') + } else { + console.log('') + const bin = buildGoHelper() + // Go does not create its ScratchRoot either. + mkdirSync(join(root, 'scratch-go'), { recursive: true }) + + // Go writes a base plus a WAL segment on top. + runGo(bin, ['write', root]) + + // Node restores the Go base and replays the Go WAL, then appends. + { + const ns = namespace(`file://${root}/ns`, 'xbind-go') + const phases = [] + const obj = await ns.open('goobj', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM g ORDER BY id', { format: 'CSV' }) + report('Node restores a Go base', phases.includes('restoring-base')) + report('Node replays a Go WAL', phases.includes('replaying-wal')) + report( + 'Go -> Node rows match', + rows === '1,"go-one"\n2,"go-two"\n3,"go-three"\n', + rows.trim().replace(/\n/g, ' | '), + ) + await obj.execute("INSERT INTO g VALUES (4, 'node-four')") + await obj.flush() + await obj.close() + } + + // Go reads Node's append back. + { + const out = runGo(bin, ['read', root, 'goobj', 'g']) + report( + 'Node -> Go rows match', + out === '1,"go-one"\n2,"go-two"\n3,"go-three"\n4,"node-four"\n', + out.trim().replace(/\n/g, ' | '), + ) + } + + // Node checkpoints; Go restores from that archive with no WAL left. + { + const ns = namespace(`file://${root}/ns`, 'xbind-go-nw') + const obj = await ns.open('nodeobj-go', { database: 'default' }) + await obj.execute('CREATE TABLE n (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO n VALUES (1, 'from-node')") + await obj.checkpoint() + report('Node checkpoint left no WAL (for Go)', obj.manifest.wal.length === 0) + await obj.close() + + const out = runGo(bin, ['read', root, 'nodeobj-go', 'n']) + report("Go RESTOREs Node's archive", out === '1,"from-node"\n', out.trim()) + } + ran++ + } + + if (ran === 0) { + console.error('\nNo other binding was available, so nothing was verified.') + failed = true + } +} finally { + rmSync(root, { recursive: true, force: true }) +} + +console.log(failed ? '\nSOMETHING FAILED' : '\nall exchanges hold') +process.exit(failed ? 1 : 0) diff --git a/scripts/durable-cross-version.ts b/scripts/durable-cross-version.ts new file mode 100644 index 0000000..fdad93d --- /dev/null +++ b/scripts/durable-cross-version.ts @@ -0,0 +1,139 @@ +/** + * Does a newer engine restore what an older one wrote? + * + * That is the compatibility promise `chdb/durable` rests on — the `min_reader` + * and `backup_format` gates exist to admit exactly this case — and until + * `26.7.3` shipped it could not be tested, because `26.7.2-rc.2` was the first + * engine with the durable ABI and there was nothing earlier to restore. + * + * Three claims, in order: + * + * 1. an object written by the OLD engine (full checkpoint plus a WAL segment + * on top) opens on the NEW one and yields the same rows; + * 2. the floor rises on that write — `min_reader` becomes the new version — + * and the OLD engine is then refused with `engine_incompatible` rather + * than left to fail partway through RESTORE; + * 3. neither direction silently half-works. + * + * Runs under Bun because the addon links one engine at build time, while + * `bun:ffi` can `dlopen` whichever it is handed. + * + * Each stage runs in its OWN process, and that is not tidiness. A libchdb is + * `dlopen`ed once per process — `loadLibchdb` caches it, and it has to, + * because the engine binds one data path per process and a second `dlopen` + * would be a second EmbeddedServer. So a single-process version of this + * script silently tests one engine twice: the first library loaded answers + * every later request, both versions report the same number, and the floor + * never rises. It passes while proving nothing. Hence the re-exec below. + * + * ```sh + * CHDB_LIBCHDB_OLD=/path/to/26.7.2-rc.2/libchdb.so \ + * CHDB_LIBCHDB_NEW=/path/to/26.7.3/libchdb.so \ + * bun scripts/durable-cross-version.ts + * ``` + * + * Not in CI: it needs two ~100 MB engine downloads, and what it checks moves + * only when core releases. Run it when adopting an engine, and record the + * result in docs/design/durable-control-plane.md. + */ + +import { mkdtemp, mkdir, rm } from 'fs/promises' +import { tmpdir } from 'os' +import { join } from 'path' + +import { DurableNamespace } from '../src/durable/namespace' +import { isDurableErrorOf } from '../src/durable/errors' +import { LibchdbFfiEngine } from '../test/durable/libchdb-ffi' + +const OLD = process.env['CHDB_LIBCHDB_OLD'] +const NEW = process.env['CHDB_LIBCHDB_NEW'] +if (!OLD || !NEW) { + console.error( + 'Set CHDB_LIBCHDB_OLD and CHDB_LIBCHDB_NEW to two libchdb builds.\n' + + 'Fetch one with: curl -L -o old.tar.gz \\\n' + + ' https://github.com/chdb-io/chdb-core/releases/download/v26.7.2-rc.2/macos-arm64-libchdb.tar.gz', + ) + process.exit(2) +} + +const ROWS = '1,"before-checkpoint"\n2,"also-before"\n3,"after-checkpoint"\n' + +/** One namespace over a shared object store, driven by the given engine. */ +async function namespaceOn(library: string, root: string, scratch: string): Promise { + await mkdir(scratch, { recursive: true }) + return new DurableNamespace(`file://${root}/ns`, { + engineFactory: () => new LibchdbFfiEngine({ libraryPath: library }), + scratchRoot: scratch, + // Long enough that a checkpoint never races the lease on a slow machine. + tuning: { leaseTtlMs: 120_000, heartbeatIntervalMs: 30_000 }, + }) +} + +function report(claim: string, ok: boolean, detail = ''): void { + console.log(`${ok ? 'PASS' : 'FAIL'} ${claim}${detail ? ` — ${detail}` : ''}`) + if (!ok) process.exitCode = 1 +} + +const phase = process.env['XVER_PHASE'] +const root = process.env['XVER_ROOT'] + +if (phase && root) { + // ---- a single stage, in its own process with exactly one engine ---- + if (phase === 'write') { + const ns = await namespaceOn(OLD, root, join(root, 'scratch-old')) + const obj = await ns.open('xver', { database: 'default' }) + await obj.execute('CREATE TABLE events (id UInt64, note String) ENGINE = MergeTree ORDER BY id') + await obj.execute("INSERT INTO events VALUES (1, 'before-checkpoint'), (2, 'also-before')") + await obj.checkpoint() + await obj.execute("INSERT INTO events VALUES (3, 'after-checkpoint')") + await obj.flush() + await obj.close() + console.log(`WROTE ${await new LibchdbFfiEngine({ libraryPath: OLD }).version()}`) + } else if (phase === 'restore') { + const ns = await namespaceOn(NEW, root, join(root, 'scratch-new')) + const phases: string[] = [] + const obj = await ns.open('xver', { onRestoreProgress: (p) => phases.push(p.phase) }) + const rows = await obj.query('SELECT id, note FROM events ORDER BY id', { format: 'CSV' }) + report('new engine restores the old base', phases.includes('restoring-base')) + report('and replays the WAL on top', phases.includes('replaying-wal')) + report('rows survive the version change', rows === ROWS, rows === ROWS ? '' : JSON.stringify(rows)) + await obj.close() + console.log(`RESTORED ${await new LibchdbFfiEngine({ libraryPath: NEW }).version()}`) + } else if (phase === 'refuse') { + const ns = await namespaceOn(OLD, root, join(root, 'scratch-back')) + const e = await ns.open('xver').then( + (o) => o.close().then(() => undefined), + (err: unknown) => err, + ) + report( + 'old engine refused once the floor rose', + isDurableErrorOf(e, 'engine_incompatible'), + e instanceof Error ? e.message.slice(0, 110) : 'it opened', + ) + } else { + console.error(`unknown XVER_PHASE ${phase}`) + process.exit(2) + } + process.exit(process.exitCode ?? 0) +} + +// ---- orchestrator ---- +const workdir = await mkdtemp(join(tmpdir(), 'durable-xver-')) +let failed = false +try { + console.log(`old: ${OLD}\nnew: ${NEW}\n`) + for (const stage of ['write', 'restore', 'refuse'] as const) { + const proc = Bun.spawnSync({ + cmd: [process.execPath, import.meta.path], + env: { ...process.env, XVER_PHASE: stage, XVER_ROOT: workdir }, + stdout: 'inherit', + stderr: 'inherit', + }) + if (proc.exitCode !== 0) failed = true + } +} finally { + await rm(workdir, { recursive: true, force: true }) +} + +console.log(failed ? '\nSOMETHING FAILED' : '\nall claims hold') +process.exit(failed ? 1 : 0) diff --git a/src/durable/backends/s3.ts b/src/durable/backends/s3.ts index eef87c4..41373fc 100644 --- a/src/durable/backends/s3.ts +++ b/src/durable/backends/s3.ts @@ -72,11 +72,11 @@ import { createReadStream } from 'fs' import { stat } from 'fs/promises' import type { Readable } from 'stream' -import { - GetObjectCommand, - PutObjectCommand, - S3Client, - type S3ClientConfig, +import type { + GetObjectCommand as GetObjectCommandCtor, + PutObjectCommand as PutObjectCommandCtor, + S3Client as S3ClientCtor, + S3ClientConfig, } from '@aws-sdk/client-s3' import type { DurableBackend, GetWithEtag, PutOutcome, ReplaceOutcome } from '../backend' @@ -84,6 +84,40 @@ import { DurableBackendError, DurableLimitExceededError } from '../errors' import { isValidObjectKey } from '../keys' import { registerBackendScheme } from '../namespace' +/** + * The AWS SDK, loaded on first use rather than on import. + * + * `@aws-sdk/client-s3` is an *optional* peer dependency — a caller using only + * the local backend should not have to install several megabytes of it. The + * cost of that is the failure a caller meets when they do want S3 and have not + * installed it, and a static import made that failure a bare + * `Cannot find module '@aws-sdk/client-s3'` thrown from a file they have never + * heard of, at `require('chdb/durable/s3')`. + * + * Deferring the load moves it to the first S3 operation and lets it say what + * to install. The rest of this module — the scheme registration, the key + * validation, the size limit — loads and works either way. + */ +type AwsS3Sdk = typeof import('@aws-sdk/client-s3') + +let sdk: AwsS3Sdk | undefined + +function awsSdk(): AwsS3Sdk { + if (sdk) return sdk + try { + // eslint-disable-next-line @typescript-eslint/no-var-requires + sdk = require('@aws-sdk/client-s3') as AwsS3Sdk + } catch (e) { + throw new DurableBackendError( + `durable: chdb/durable/s3 needs @aws-sdk/client-s3, which is an optional peer ` + + `dependency so that callers using only the local backend do not pay for it. ` + + `Install it: npm install @aws-sdk/client-s3`, + { cause: e }, + ) + } + return sdk +} + /** Ceiling for a single PutObject. Beyond this a checkpoint needs multipart. */ export const MAX_SINGLE_PUT_BYTES = 5 * 1024 * 1024 * 1024 @@ -92,7 +126,7 @@ export interface S3BackendOptions { /** Key prefix for this object, without a leading slash. May be empty. */ prefix?: string /** Pre-built client. Takes precedence over `clientConfig`. */ - client?: S3Client + client?: S3ClientCtor /** Passed to `new S3Client`. Credentials come from the default chain unless set. */ clientConfig?: S3ClientConfig } @@ -177,7 +211,7 @@ function backendError(what: string, describe: string, e: unknown): DurableBacken export class S3DurableBackend implements DurableBackend { readonly describe: string - private readonly client: S3Client + private readonly client: S3ClientCtor private readonly bucket: string private readonly prefix: string @@ -185,7 +219,7 @@ export class S3DurableBackend implements DurableBackend { if (!options.bucket) throw new RangeError('durable: S3 backend requires a bucket') this.bucket = options.bucket this.prefix = options.prefix ? options.prefix.replace(/^\/+|\/+$/g, '') : '' - this.client = options.client ?? new S3Client(options.clientConfig ?? {}) + this.client = options.client ?? new (awsSdk().S3Client)(options.clientConfig ?? {}) // Never the endpoint's credentials, and never a presigned anything: this // string ends up in error messages and logs. this.describe = `s3://${this.bucket}/${this.prefix}` @@ -206,7 +240,7 @@ export class S3DurableBackend implements DurableBackend { async getBytesWithEtag(key: string): Promise { try { const out = await this.client.send( - new GetObjectCommand({ Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().GetObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key) }), ) if (!out.Body) return undefined const bytes = await out.Body.transformToByteArray() @@ -230,7 +264,7 @@ export class S3DurableBackend implements DurableBackend { async openReadStream(key: string): Promise { try { const out = await this.client.send( - new GetObjectCommand({ Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().GetObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key) }), ) if (!out.Body) return undefined // Under Node the SDK hands back a Readable; the union also covers the @@ -268,7 +302,7 @@ export class S3DurableBackend implements DurableBackend { this.assertWithinSinglePut(key, bytes.byteLength) try { const out = await this.client.send( - new PutObjectCommand({ + new (awsSdk().PutObjectCommand)({ Bucket: this.bucket, Key: this.keyFor(key), Body: bytes, @@ -291,11 +325,11 @@ export class S3DurableBackend implements DurableBackend { private async conditionalPut( key: string, - input: Omit[0], 'Bucket' | 'Key'>, + input: Omit[0], 'Bucket' | 'Key'>, ): Promise { try { await this.client.send( - new PutObjectCommand({ ...input, Bucket: this.bucket, Key: this.keyFor(key) }), + new (awsSdk().PutObjectCommand)({ ...input, Bucket: this.bucket, Key: this.keyFor(key) }), ) return 'created' } catch (e) { diff --git a/src/durable/namespace.ts b/src/durable/namespace.ts index 250e853..9d00f99 100644 --- a/src/durable/namespace.ts +++ b/src/durable/namespace.ts @@ -57,13 +57,57 @@ export function registerBackendScheme(scheme: string, factory: BackendSchemeFact SCHEMES.set(scheme.replace(/:$/, ''), factory) } -registerBackendScheme('file', (url, objectId) => { - const root = fileURLToPath(url) +/** + * The local filesystem, under two scheme names. + * + * `file:` is this binding's spelling; `local:` is Python's, and Go registers + * the same backend under both for the same reason it matters here — a + * namespace URL is the sort of thing that ends up in a config file shared + * between a Python service and a Node one, and refusing the other binding's + * spelling turns that into a support ticket. Node was the only one of the + * three that took just one. + * + * The two forms differ in how the path is written, not in where it points. + * `file:` is a real URL, so a space is `%20`; `local:` is opaque, so the path + * is closer to literal. Beyond ordinary paths the bindings do not agree + * anyway (Python leaves `local:` undecoded, Go decodes it), so this follows + * Go — one factory, decode both — rather than inventing a third behaviour for + * a case none of them share. + */ +const localFilesystem: BackendSchemeFactory = (url, objectId) => { + const root = url.protocol === 'file:' ? fileURLToPath(url) : opaquePath(url) if (!isAbsolute(root)) { - throw new RangeError(`durable: file namespace must be an absolute path, got ${url.href}`) + throw new RangeError( + `durable: a ${url.protocol} namespace must be an absolute path, got ${url.href}`, + ) } return new LocalDurableBackend({ root: join(root, objectId) }) -}) +} + +/** The path inside a `local:` URL, which is not a `file:` URL and has no host. */ +function opaquePath(url: URL): string { + // A host would be silently dropped otherwise, and `local://data/objects` far + // more likely means a mistyped path than a hostname nobody can resolve. + if (url.host) { + throw new RangeError( + `durable: a ${url.protocol} namespace takes a path, not a host; got ${url.href}. ` + + `Write local:/absolute/path`, + ) + } + try { + return decodeURIComponent(url.pathname) + } catch { + // A stray `%` that is not an escape. Reported here rather than left as a + // bare URIError, which says nothing about which input produced it. + throw new RangeError( + `durable: ${url.href} has a malformed percent-escape in its path; write a literal ` + + `'%' as '%25', or use a file: URL`, + ) + } +} + +registerBackendScheme('file', localFilesystem) +registerBackendScheme('local', localFilesystem) export interface DurableNamespaceOptions { /** How to build the engine for an object being opened. Required. */ diff --git a/test/durable/protocol.test.ts b/test/durable/protocol.test.ts index 52b5a38..c1315ac 100644 --- a/test/durable/protocol.test.ts +++ b/test/durable/protocol.test.ts @@ -13,6 +13,7 @@ import { existsSync } from 'fs' import { chmod, mkdir, mkdtemp, readFile, writeFile } from 'fs/promises' import { tmpdir } from 'os' import { join } from 'path' +import { pathToFileURL } from 'url' import { coldHead, parseHead, serializeHead } from '../../src/durable/head' import { assertEngineCompatible, assertReadable, assertWritable } from '../../src/durable/negotiate' @@ -21,6 +22,8 @@ import { decodeWalSegment, encodeWalSegment, walLineBytes } from '../../src/dura import { checkpointKey, isValidObjectKey, walKey } from '../../src/durable/keys' import { LIMITS } from '../../src/durable/types' import { LocalDurableBackend } from '../../src/durable/backends/local' +import { DurableNamespace } from '../../src/durable/namespace' +import { FakeEngine } from './fakes' import { QueryClass, assertExecuteAllowed, @@ -770,3 +773,54 @@ describe('local backend conditional operations', () => { expect(await be.openReadStream('checkpoints/nope.tar.gz')).toBeUndefined() }) }) + +describe('namespace URL schemes', () => { + // A namespace URL is the sort of thing that ends up in a config file shared + // between a Python service and a Node one. Python spells the local backend + // `local:`, Go registers both, and this binding used to take only `file:` — + // so one config worked in two bindings out of three. + const engineFactory = () => new FakeEngine() + + /** Open and return the failure, which is how the resolved path surfaces. */ + async function openMissing(url: string): Promise { + const ns = new DurableNamespace(url, { engineFactory }) + try { + await ns.open('missing', { readOnly: true }) + throw new Error(`expected ${url} to report a missing object`) + } catch (e) { + return e as Error + } + } + + it('accepts both spellings, and resolves them to the same place', async () => { + const root = await mkdtemp(join(tmpdir(), 'scheme-')) + const dir = join(root, 'ns') + // not_found rather than a scheme error means the backend resolved and + // looked: the object really is absent. + const viaFile = await openMissing(pathToFileURL(dir).href) + const viaLocal = await openMissing(`local:${dir}`) + expect(isDurableErrorOf(viaFile, 'not_found')).toBe(true) + expect(isDurableErrorOf(viaLocal, 'not_found')).toBe(true) + // The message names the resolved location, so equal messages mean both + // spellings point at one directory — which is the whole claim. + expect(viaLocal.message).toBe(viaFile.message) + }) + + it('takes a path containing a space through the local spelling', async () => { + // `file:` percent-encodes a space; `local:` carries it literally, and a + // decoded path is what the other bindings resolve to. + const root = await mkdtemp(join(tmpdir(), 'scheme with space-')) + expect(isDurableErrorOf(await openMissing(`local:${join(root, 'ns')}`), 'not_found')).toBe(true) + }) + + it('refuses a local URL naming a host instead of a path', async () => { + // Dropping the host silently would resolve `local://data/objects` to + // `/objects`, which is not what anyone wrote. + expect((await openMissing('local://data/objects')).message).toMatch(/takes a path, not a host/) + }) + + it('refuses a relative path and a malformed percent-escape', async () => { + expect((await openMissing('local:relative/path')).message).toMatch(/absolute path/) + expect((await openMissing('local:/tmp/100%pct')).message).toMatch(/percent-escape/) + }) +}) diff --git a/test/durable/s3-backend.test.ts b/test/durable/s3-backend.test.ts index c0b9523..ef54d7f 100644 --- a/test/durable/s3-backend.test.ts +++ b/test/durable/s3-backend.test.ts @@ -179,6 +179,11 @@ describe.skipIf(!BUCKET)(`S3-compatible backend (${ENDPOINT ?? 'aws'})`, () => { }) describe('large objects', () => { + // 180s rather than the suite's 20s. 12 MiB is nothing against a local + // MinIO — 0.4s — and about a minute against a real bucket across an + // ocean, so the default made this the one test that fails on AWS while + // the implementation is fine. A red suite that means "your network is + // far away" trains people to ignore it. it('streams a file up and back down with its digest intact', async () => { const be = backend('stream') const dir = await mkdtemp(join(tmpdir(), 'durable-s3-')) @@ -206,7 +211,7 @@ describe.skipIf(!BUCKET)(`S3-compatible backend (${ENDPOINT ?? 'aws'})`, () => { }) expect(size).toBe(expected.size) expect(hash.digest('hex')).toBe(expected.sha256) - }) + }, 180_000) }) describe('recovery on another machine', () => { diff --git a/test/durable/subpath.test.ts b/test/durable/subpath.test.ts index 96031e5..904815d 100644 --- a/test/durable/subpath.test.ts +++ b/test/durable/subpath.test.ts @@ -131,6 +131,42 @@ describe.skipIf(!built)('subpath imports', () => { expect(out).toContain("chdb/durable/s3") }) + it('loads chdb/durable/s3 even without the AWS SDK, and says what to install', () => { + // The SDK is an optional peer dependency, so a caller using only the local + // backend does not pay for it. That makes "I do want S3 and forgot to + // install it" a real path, and a static import made it a bare + // `Cannot find module '@aws-sdk/client-s3'` from a file the caller has + // never heard of. Checked in a child process with the module resolution + // broken, since the SDK is installed in this tree. + const out = runNode(` + const Module = require('module') + const real = Module._resolveFilename + Module._resolveFilename = function (request, ...rest) { + if (request === '@aws-sdk/client-s3') { + const e = new Error("Cannot find module '@aws-sdk/client-s3'") + e.code = 'MODULE_NOT_FOUND' + throw e + } + return real.call(this, request, ...rest) + } + // The module itself still loads, and still registers the scheme. + const s3 = require('./dist/durable/backends/s3.js') + if (typeof s3.S3DurableBackend !== 'function') throw new Error('no S3DurableBackend') + const { DurableNamespace } = require('./dist/durable/index.js') + new DurableNamespace('s3://b/p?region=us-east-1', { engineFactory: () => ({}) }) + // Only constructing a backend reaches the SDK. + try { + new s3.S3DurableBackend({ bucket: 'b', prefix: 'p' }) + throw new Error('expected a missing-dependency failure') + } catch (e) { + if (!e.message.includes('@aws-sdk/client-s3')) throw new Error('unhelpful: ' + e.message) + if (!e.message.includes('npm install')) throw new Error('no fix given: ' + e.message) + console.log('clean') + } + `) + expect(out).toBe('clean') + }) + it('declares every durable subpath in package.json exports', () => { const pkg = require('../../package.json') as { exports: Record } expect(pkg.exports['./durable']).toBeDefined()