Skip to content

fix(runtime): mitigate shared SQLite write contention - #163

Merged
kingsword09 merged 2 commits into
kingsword09:mainfrom
mwotton:fix/sqlite-busy-timeout
Sep 19, 2026
Merged

kingsword09 merged 2 commits into
kingsword09:mainfrom
mwotton:fix/sqlite-busy-timeout

Conversation

@mwotton

@mwotton mwotton commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Related to #162. This is a bounded contention mitigation, not a claim that sustained concurrent-write failures are fully resolved. Keep #162 open for reporter validation and upstream follow-up.

Why the original patch needed adjustment

The reporting host runs multiple headless agents and interactive sessions against one shared SQLite session database. Lock errors can escape as fatal Turn execution failed errors.

The currently pinned 3.12.3 runtime already opens SQLite with a 5-second timeout. Its asynchronous startup migration temporarily uses a shorter busy timeout and restores 5000 ms in finally. Setting 10000 ms immediately after new DatabaseSync(...) is therefore overwritten on the normal openStartup() path. Merely finding a pragma string in the bundle is not evidence that the active connection uses it.

Changes

  • Apply the 10000 ms ordinary-write timeout after successful migrations on both synchronous construction and asynchronous startup. Preserve startup migration budgets, backoff, rollback, and failure cleanup.
  • Keep the patch required and idempotent; verify both lifecycle insertion points rather than accepting an unrelated pragma string.
  • Pin Bun and bun-types to 1.4.1 in development and release workflows.
  • Replace the Bun-executed SQLite driver check with five real Node tests using the native vendored SqliteSessionStore, temporary databases, and independent Node processes.
  • Build and pack once in CI, then test the exact validated artifact with Node 22.19.0, 24, and 26 on Linux and Node 24 on macOS.
  • Document the reproduction and remaining limits in docs/SQLITE_CONCURRENCY.md.

Local validation

  • TypeScript checking, locked runtime synchronization, runtime/package checks, terminal smoke tests, and tarball installation smoke passed.
  • Bun 1.4.1: 629 unit tests, 43 TUI tests, and 21 runtime integration tests passed.
  • All five Node SQLite tests passed against the packed artifact on Node 22.19.0, 24.21.0, and 26.4.0:
    1. effective timeout after synchronous open, asynchronous startup, and reopen;
    2. a native session write succeeds exactly once after an independent writer holds its lock for 6.5 seconds;
    3. a deliberately short timeout reports SQLITE_BUSY without writing a session, and the connection works after release;
    4. four independent processes migrate one fresh database and preserve all session writes;
    5. a killed writer's uncommitted changes do not survive reopen, and later writes succeed.

These are deterministic storage-level regressions, not a reproduction of every detail of the reporter's production fleet. Hosted CI must pass before merge.

Scope and follow-up

No schema changes, historical-data rewrite, automatic whole-turn retry, database sharding, or Desktop 3.14.0 runtime upgrade are included. A lock held beyond the timeout can still fail, and synchronous SQLite waiting can block the calling runtime's event loop.

After release, ask the reporter to test the same concurrent workload and share sanitized lock/turn-failure evidence. Upstream ZCode should investigate long transactions, contention diagnostics, and safe persistence-boundary recovery; blindly replaying an entire agent turn can duplicate external side effects.

mwotton and others added 2 commits September 19, 2026 06:38
…busy

Concurrent zcode processes share ~/.zcode/cli/db/db.sqlite. The vendored
runtime opens it with node:sqlite, whose default busy timeout is 0, so
write contention at a turn boundary surfaces as an immediate SQLITE_BUSY
("database is locked") that fatally fails the turn (refs kingsword09#162).

Inject `pragma busy_timeout = 10000` at the SqliteSessionStore connection
site (the only DatabaseSync construction in the bundle) through the
existing runtime patch framework, so writers wait for the lock instead of
dying. The migrations' own WAL-mode retry budget is unchanged.

Follow-up for the upstream runtime: treat SQLITE_BUSY at the persistence
layer as retryable with backoff, since a busy timeout only bounds the
wait.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@kingsword09 kingsword09 changed the title fix(runtime): set busy_timeout on the shared session DB so writers wait instead of dying fix(runtime): mitigate shared SQLite write contention Sep 19, 2026
@kingsword09
kingsword09 merged commit 16c12c1 into kingsword09:main Sep 19, 2026
5 checks passed
seanchatmangpt referenced this pull request in seanchatmangpt/zcode-cli Sep 23, 2026
…14.0/3.14.1 line)

Hand-resolved conflicts, both sides read; no -X ours/theirs. Upstream
skips origin/release/zcode-upstream 05d3f49 (superseded).

- package.json: upstream adds test:node (node --test test/node/*.test.cjs,
  the #163 sqlite-session-store regression test) and test:all runs it;
  ours keeps ZCODE_REQUIRE_TOOLCHAINS=1 strict gates (ZOCEL-003) on
  test:unit/test:runtime and build = node node_modules/.bin/tsdown
  (release-package assertion). Both sides' release-package assertions are
  merged and pass against this resolution.
- scripts/sync-runtime.ts: upstream's queryHelper-conditional CLI mode
  bridge adopted, keeping our replaceWarmed hardening; upstream's
  label-anchored runStreamText detection (3.14.1-compatible) adopted over
  our positional pattern; patch registry keeps BOTH our optional metadata
  null-guards AND upstream's required sqlite-busy-timeout (#163 shared
  SQLite write contention fix for concurrent workers).
- test/runtime/launcher.test.ts: both env fixtures kept (hermetic-env.ts
  for credential blanking, upstream runtime-env.ts for ZCODE_ stripping +
  provider-config pinning); keyless-prompt test merges both override sets.
- Runtime relock: lock and package carry upstream's 3.14.1 / 3.14.1-27
  (per release action: upstream semver, no CalVer tag).
seanchatmangpt referenced this pull request in seanchatmangpt/zcode-cli Sep 23, 2026
… re-anchors

The 3.14.1 runtime (sha512-verified against zcode-runtime.lock.json)
changed minified shapes and event wiring; upstream's own patch plan at
dc82485 fails 8 required patches against its own pinned artifact. This
ports the fork's plan onto the real bundle:

- scripts/sync-runtime.ts:
  * session-model-recovery: all three anchors via execWarmed (a cold
    regex exec spuriously misses on the multi-MB 3.14.1 bundle -- every
    anchor of this patch missed under Bun/JSC).
  * max-turns-enforcement: the runRegularTurnLoop label anchor is
    helper-agnostic (3.14.x renamed the registration helper a(...) -> r(...)).
  * streaming-ledger-forwarding: 3.14+ consumes streaming ledger events
    natively (projection-table handler entry) -> typed no-op.
  * tui-bridge skill-list host: 3.14 attaches app queries in a helper
    whose submitter call site moved -> degrade to the host-less bridge
    method instead of refusing the whole bridge.
  * model-catalog-reload: 3.14+ owns catalog reload natively in the
    registry service -> typed no-op (verify accepts the native shape).
- scripts/zcode-events.ts: internal event enum and wire projection found
  structurally (3.12.3 W/wCs renamed in 3.14.1); grouped fall-through
  cases wired correctly.
- ontology/zcode-loop.ttl: + Internal-DynamicWorkflowRunProgress
  (new 3.14.1 internal event); src/generated regenerated (fixed point).
- test/mcp-user-scope-registration.test.ts: anchors are structural
  regexes (the xaas worker .mcp.json law survives minifier renames).
- test/capability-snapshot.test.ts: temp root via realpath(tmpdir()) --
  bsdtar refuses to extract archive members through the /var symlink.
- test/gall-work-permission.test.ts: environmental deferral typed on
  provider 1302 throttle / sandboxed model-catalog windows.

Gate results on the 3.14.1 tree: sync:locked rc=0 (all required patches
apply+verify; anchor-drift + loop-gaps 9 pass); typecheck 0; test:node 5
pass (incl. #163 sqlite-session-store); test:unit 789 pass with the live
permission qualification deferred (environmental); test:runtime 9 fail
vs 18 on the pre-merge tree (composition: sandbox-bound TUI/resume
smokes + the two pre-existing subagent-harness failures; every
new-anchor port itself is proven by sync:locked).
seanchatmangpt referenced this pull request in seanchatmangpt/zcode-cli Sep 23, 2026
… relock

Upstream main merged (6 commits incl. #163 SQLite contention fix,
3.14.0/3.14.1 protocol+TUI bridges, #169 release validation); release
superseded 05d3f49 skipped. Lock/vendor/package on the 3.14.1 /
3.14.1-27 line per the release action (upstream semver, no CalVer tag).
Runtime anchors ported to the 3.14.1 bundle and re-proven by
sync:locked (anchor-drift + loop-gaps green: maxTurns, warm-up).
Standing PARTIAL_ALIVE: test:runtime retains 9 sandbox-bound failures
(18 on the pre-merge tree); live permission leg deferred (environmental).
seanchatmangpt referenced this pull request in seanchatmangpt/zcode-cli Sep 23, 2026
…line

Upstream kingsword09 dc82485 (incl. #163 SQLite contention fix), fork/main
hand-resolved per decision #4, fin/* + headless-mcp contract, native
gall-work lifecycle (release side kept in the add/add: strict superset
carrying WO3 --mode yolo posture, WO6 lease-identity OCEL binding, WO10
provider capacity backoff), launcher wiring with gall dispatch first +
maxTurns, provider capacity backoff, OCEL receipt identity binding,
hygiene (PRs #1/#2/#3 retired, worktrees 21 -> 3, dead tracking refs
pruned), v26.9.21 ZOCEL-010 exact-head receipt. Runtime relocked to
3.14.1 with anchors ported and re-proven (sync:locked green).
Receipts under receipts/v26.9.22/.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants