Skip to content

fix(sessions)!: scope stale session-store refusals and reset - #2613

Merged
ScriptedAlchemy merged 2 commits into
masterfrom
fleet/scoped-store-reset
Sep 29, 2026
Merged

ScriptedAlchemy merged 2 commits into
masterfrom
fleet/scoped-store-reset

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Cause

After beta.61 the operator's profile-sessions store and two project-sessions stores still held observation rows from before the unified observation identity. Schema admission refused the whole store (ResetRequired { authority: "observations" }). A project-sessions store also holds the project's configuration and query cursor keys, so the project's own open failed. Project open landed in Stalled (UnrepairableVerdict) and scheduled another retry. The core route never published, so status, search, and every other code tool answered application.surface.reset_required, and the code graph never published either. Doctor counted the stall as an issue and printed Run tracedecay install. The named remedy was tracedecay wipe --all --yes.

A second, separate break: since #2332 the retained core route never mounted its query authority after a daemon restart. That route is the one serving whenever a session store refuses the full upgrade. The waiter polled latest_generation_id, and a partitioned restore leaves that empty, so search stayed graph_warming. The existing reset_required_profile_session_store_is_served_typed_until_its_named_reset journey already fails on this.

Design

Refusal scoped to the stale authority.

  • ensure_observation_schema reports stale observation rows as a refused authority instead of failing the whole store. The rows stay unread, the marker is not adopted, and nothing is rewritten. The store's other authorities are admitted, it is never converged, and the owner and leases carry the refusal (reset_required()).
  • The session registry fences session features. project_sessions, mount_registered_project_sessions, mounted_project_sessions, and profile_sessions answer the typed refusal. project_session_store and mounted_project_session_store serve only the non-session authorities (configuration and query cursor keys). Project open, branch-admin configuration, and the query-authority mount use those.
  • When a reset-required session store refuses the full upgrade, the project runtime publication records ResetRequired(detail). Owners that are still missing then answer that typed refusal (kind: reset_required, remedy tracedecay wipe --stale --yes) instead of owner_failed. The retained core also registers the code-read owners (callable-code authorization and primitive runtime). Callers, file dependents, and diagnostics serve, and session lookups answer TemporalRetrievalFailure::ResetRequired.
  • The core route's query-authority waiter now reuses retry_deferred_query_authority_until_serving, the same wait the full route's deferred mount already uses. The hand-written loop that missed partitioned restores is deleted.

Scoped remedy: tracedecay wipe --stale --yes. It is the only new shape.

  • It reads the reset census from the running daemon, which is the authority that refused the stores, and parses each store label through ResettableStoreV1.
  • It refuses up front, deleting nothing, if any listed store cannot be reset on its own. The profile authority's remedy stays wipe --all --yes.
  • It takes the profile offline. A managed service is quiesced and restored afterwards; for an unmanaged daemon it waits for the operator to stop it.
  • It deletes exactly the refused stores' families: <name>.db*, .<name>.db.*, <name>.grafeo*. There is no backup or copy, and the daemon recreates each store empty.
  • Session-store StoreResetRequiredV1.remedy, tool refusals for registered-store authorities, doctor, and update now name tracedecay wipe --stale --yes.
  • Doctor reports each reset store as a pending operator action (counted in pending_actions), not a warning.

Fails on master

The new typed_terminal_restart_acceptance::stale_sessions_store_reset::stale_session_stores_refuse_sessions_only_until_their_scoped_reset fails on f883da9:

panicked at .../stale_sessions_store_reset.rs:210:9:
code search never answered `probe`: {... "problem":{... "code":"application.surface.reset_required", ... "message":"The application store requires an explicit reset" ...}}

The existing reset_required_serving::reset_required_profile_session_store_is_served_typed_until_its_named_reset fails with master's query-authority waiter:

code index never answered `probe` within 120s: {... "reason":"graph_warming" ... served_generation=none ...}

Both pass on this branch. The new test covers:

  • MCP initialize and tools/list, search, callers, and file dependents serving over stale stores.
  • The exact reset census.
  • Project open not stalled.
  • Project and user session reads refused as typed reset_required naming tracedecay wipe --stale --yes.
  • Doctor naming that command, with no issues and no install footer.
  • wipe --stale deleting only the two session stores. Every other profile file stays byte-identical except lifecycle.lock.
  • Sessions serving from fresh stores afterwards.

Runtime journey (built tracedecay binary)

Setup: an isolated HOME, a clone of cargo-slot as the corpus, and a daemon under a 6 GB systemd-run scope. The corpus was indexed, the daemon stopped, and both session stores given the pre-unified observation row with the marker removed. Then the daemon was restarted:

tool search choose_slot        -> "coverage":{"exact":"complete","graph":"complete","lexical":"complete","recall":"full"}
tool status                    -> reset_required_stores: profile sessions + project sessions proj_5620e52c93b373e4,
                                  authority "observations", remedy "tracedecay wipe --stale --yes"; project_open: null
tool callers / file_dependents -> "outcome":"evidence"
tool diagnostics               -> kind "unsupported" (no producer for a Rust-only root, same as a healthy profile)
tool lcm_status (project)      -> "kind":"reset_required","legal_actions":["reset"],
                                  detail {"authority":"observations","remedy":"tracedecay wipe --stale --yes"}
serve: initialize error None; tools/list 230 tools incl. search/callers/lcm_status
doctor (exit 75): Store profile sessions requires reset (...). Pending operator action: run `tracedecay wipe --stale --yes`
                  Store project sessions proj_5620e52c93b373e4 requires reset (...). Pending operator action: run `tracedecay wipe --stale --yes`
                  2 pending operator action(s), 4 warning(s), no issues.
wipe --stale --yes: An unmanaged TraceDecay daemon holds the profile; stop it and wipe --stale continues (waiting up to 300s).
                  reset profile sessions (4 entries removed from .../home/.tracedecay); the daemon recreates it empty
                  reset project sessions proj_5620e52c93b373e4 (5 entries removed from .../projects/proj_5620e52c93b373e4); the daemon recreates it empty
                  exit 0; tracedecay.db, tracedecay.grafeo, code-index-v1, store_manifest.json untouched
restart: search serves the same generation (no reindex); lcm_status project/user -> "outcome":"evidence"; reset_required_stores: []

Verification (local, after rebase on a7200ad)

  • cargo clippy -p tracedecay-domain -p tracedecay-global-db -p tracedecay-store-runtime -p tracedecay-daemon-service -p tracedecay-mcp -p tracedecay -p tracedecay-cli -p tracedecay-daemon-control -p tracedecay-project --all-targets --features tracedecay/test-transport,tracedecay/test-helpers,tracedecay-cli/test-transport -- -D warnings: clean. cargo fmt --all -- --check: clean.
  • mcp_suite: 607 passed.
  • transport_acceptance_suite: 14 passed, including both reset journeys. 1 failed: v2_surface_mount_conformance, because ast_grep_rewrite is unlisted when the hermetic test PATH has no ast-grep. This is environmental on this host and unrelated.
  • tracedecay-application: lib 482, application_suite 66, pr_tracking 7.
  • tracedecay-daemon-service: 322 (includes adoption_observation). tracedecay-global-db: 385. tracedecay-mcp: 387. tracedecay-domain: 224 plus domain_suite 168. tracedecay-store-runtime: 122. tracedecay-daemon-control: 100. tracedecay-project: 38.
  • CLI bin (tracedecay): 347 passed.
  • tracedecay lib: 768 passed. 1 failed: verified_graph_query_authority_tests::absent_query_port_fails_closed_for_every_awaiting_graph_handler. It is red on master since feat(loom)!: record session-attributed test runs durably #2583, reproduced on f883da9 and filed as absent_query_port_fails_closed_for_every_awaiting_graph_handler red since #2583 #2611.

Operator command after the next release

tracedecay wipe --stale --yes

Fixes #2587

BREAKING CHANGE: session-store reset refusals name tracedecay wipe --stale --yes; wipe gains --stale.

@changeset-bot

changeset-bot Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bd49607

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-29T09:50:18.613182Z d6dbe50 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6dbe50a22

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +271 to +272
session_store,
Arc::new(primitive_runtime::ResetRequiredSessionLookupV1),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Refuse observation-backed primitives on stale stores

When the project session store contains pre-unified observation rows, this passes its unrestricted reset-required lease into the normal primitive runtime. That runtime retains the lease as both observation_database and test_runs.store: health_delta calls compute_verified_health_delta, which can read and insert observations and converts failures into generic unavailable evidence, while test_results still queries the same session store. These operations therefore bypass the typed reset refusal—and may write into the incompatible authority—even though only code reads should remain available; use a reduced capability or explicitly refuse every observation/session-backed primitive.

AGENTS.md reference: AGENTS.md:L189-L190

Useful? React with 👍 / 👎.

A session store whose observation rows predate the unified observation
identity is admitted for its other authorities and refuses only session
features, so project open, code intelligence, and MCP keep serving.
`tracedecay wipe --stale --yes` deletes exactly the stores the daemon
reports as requiring reset.

BREAKING CHANGE: session-store reset refusals name
`tracedecay wipe --stale --yes` instead of `tracedecay wipe --all --yes`.
@ScriptedAlchemy
ScriptedAlchemy merged commit 8a16c0e into master Sep 29, 2026
3 checks passed
ScriptedAlchemy added a commit that referenced this pull request Sep 29, 2026
A session store whose LCM schema version, git correlation version, or
workflow schema identity this binary refuses is now admitted untouched
for its other authorities, the same way #2613 admits a store whose
observation rows predate the unified identity. Project open, code
intelligence, and MCP initialize/tools/list keep serving; every session
feature answers the typed reset refusal naming
`tracedecay wipe --stale --yes`; doctor counts the store as a pending
operator action; and the scoped reset deletes exactly that store.

When a later authority also fails, the earliest session refusal stays
the store's hard verdict, so a legacy store keeps its LCM identity.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Project open retries an observations reset and the graph never publishes

1 participant