v3.4.12 — Self-healing: run smooth under a running daemon (#258, #257)#259
Merged
Conversation
token gain opened token.redb eagerly at the top of run(), before dispatching any mode. redb is single-process, so a running onebrain daemon (the token.redb owner) made every mode hard-error with 'Database already open. Cannot acquire lock.' — even --history, which only reads the JSONL raw log. Fix (lazy open + daemon routing): - The default summary, --by, --history and --reset read the lock-free JSONL source of truth and never open redb → they work while a daemon holds the lock. - --all-time / --since route through the daemon's existing GET /api/token/gain when a warm vault-bound daemon holds the DB, else open Direct. - --rebuild (a redb write) and a genuinely contended Direct open now surface an actionable message (retry after `onebrain daemon stop`) instead of the raw redb error. Tests: lock-held default/history succeed; Direct-under-lock reports the actionable message; --all-time routes through a fake daemon (local redb held locked so it can ONLY pass by routing); a 404 gain route is actionable. Closes #258
Review + self-healing hardening on top of the #258 lazy-open fix: - Gap 4 (upgrade-without-restart): route --all-time/--since to a same-vault daemon REGARDLESS of version via new discover_same_vault_any_version — the gain route returns a version-stable PivotResult, so a still-running old daemon holding token.redb no longer forces a Direct open into its lock. Read verbs never restart another session's daemon (that stays ensure_running/mcp's job). - Gap 8 (daemon dies mid-call): a transport error from the gain route now falls through to a now-unlocked Direct open instead of erroring. - R2#1: a genuinely contended rollup open (and a too-old daemon route) now map to CoreError::EngineBusy -> E_ENGINE_BUSY / exit 77, the same transient-lock code search query/vsearch/get use (was a plain exit-1 error). - R2#3: fix the stale 'no live traffic writes GainEvents yet' module comment. Tests: version-skew routing, daemon-gone->Direct, by/since forwarding, rebuild-under-lock exit-77, and discover_same_vault_any_version adoption. Copilot autoreview findings (bundled): - doctor_integration find_scoped_key_line: segments[0]/last().unwrap() -> first()?/last()? (no panic on empty segments) - onebrain-fs init test: remove dead Rc<RefCell> count var + unused imports - token rollup: DateTime::<Utc>::from_timestamp -> DateTime::from_timestamp (chrono 0.4.23 deprecation)
Copilot: the token_check + token_gain test fake daemons read at most one 8192-byte chunk, so a request split across TCP segments could be misidentified. Accumulate until `\r\n\r\n` (end of HTTP headers) before routing.
…one (#258) serve was passive: it reused a version+vault-matched daemon, else ran a FOREGROUND standalone server with hold_engine=false — which (a) has no token cache, so the Token-Gain dashboard is dark (#257), and (b) collides on the redb lock / port with a different-version daemon. Now serve uses ensure_running (the confirmed reuse-or-start policy): reuse a running daemon for this vault, else start one — restarting a stale/mismatched one — then hand the user that daemon's webui URL. The started daemon holds the engine + token cache, so the dashboard is populated. A genuine start failure falls back to standalone rather than erroring. Explicit --port/--dir/$ONEBRAIN_BIND and $ONEBRAIN_NO_DAEMON still force standalone.
…te (#257, #258) Gap 1 (#257): standalone serve (the explicit --port/--dir escape hatch) opens its token.redb cache via a new ServeConfig.open_token_cache flag, so /api/token/* answers 200 and the Token-Gain dashboard lights up instead of 503-ing. The daemon (hold_engine) already opened it; the unit-test router leaves it off. Gap 3 (#258): add a vector-only 'vec' mode to /api/vault/search (run_vec_held + run_vec, mirroring the hybrid pair via Engine::vector_search), and route the CLI 'search vsearch' through it (route_to_daemon -> run_query_via_daemon 'vec'). So vsearch works while an mcp session holds the engine instead of failing E_ENGINE_BUSY. Semantic-build-gated; lex-only has no vec route (opens direct). CHANGELOG: v3.4.12 reframed to the self-healing theme. 2627 tests pass; default + lex-only builds clean; fmt + clippy clean.
Adds a download-free empty-index test for run_native's mode=vec path (run_vec), and a vec assertion to run_native_no_index_returns_empty — closing the coverage gap on the new per-request vector path.
…t only reused) Review follow-up: serve now starts the daemon via ensure_running, so the old 'Daemon already serving this vault' banner was inaccurate on the just-started path.
…258, #257) Review #2 (consistency) caught doc/comment drift from the mechanism changes: - serve.md / daemon.md / serve.rs module doc: 'serve never starts a daemon' is now false — serve reuses-or-starts (ensure_running). - daemon.md / mcp.rs / server/search.rs docs: vsearch is now daemon-routed via a new mode=vec; the route speaks lex|hybrid|vec. - ADR 0032 records the self-healing decisions and supersedes ADR 0023's 'vsearch not routed' + the old serve 'never starts a daemon' stance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v3.4.12 — Self-healing: run smooth under a running daemon
Theme: no command,
serve, or MCP call should error or fail to run because a daemon is (or isn't) holding the single-process redb lock. Started as atoken gainhotfix (#258) and grew to cover the whole daemon-contention seam.What's fixed
token gainunder a running daemon — the reported bug. Every mode hard-erroredDatabase already open. Cannot acquire lock.because the rollup DB was opened eagerly. Now the default summary /--by/--history/--resetread the lock-free JSONL source of truth, and--all-time/--sinceroute through the daemon's/api/token/gain.token gainworks right after this release without manually restarting the daemon. Newdiscover_same_vault_any_version.E_ENGINE_BUSY/ exit 77, like every sibling search surface.servereuse-or-start (Gap 2, v3.4.12: Token Gain dashboard is dark underonebrain serve(no token cache) — open token.redb read-only #257) —servenow reuses or starts a daemon (restarting a stale/version-mismatched one) instead of an engine-less foreground standalone, so the Token-Gain dashboard is populated. The explicit--port/--dirstandalone escape hatch also opens its token cache now.vsearchdaemon-routable (Gap 3) — new vector-onlymode=vecon/api/vault/search;search vsearchroutes through it instead of failingE_ENGINE_BUSYduring anonebrain mcpsession.doctorscoped-key panic guard, dead test code, achronodeprecation, and a fake-daemon read-loop.Verification
2627tests pass; default + lex-only builds clean;fmt+clippyclean.Closes #258
Closes #257