fix(daemon): retention maintenance livelocks on a stale cold-store cursor — same 48 bodies rescanned forever, daemon pinned at ~110% CPU

Summary

The daemon's retention maintenance loop got stuck in a livelock against a stale cold-store cursor: every pass rescanned the same batch of 48 canonical bodies, deleted nothing, reported has_more=true, and immediately re-ran. The cursor never advanced and never persisted, so the loop ran indefinitely and the store-thread pool stayed saturated with DB reads — daemon pinned at ~90–110% CPU for hours (survived multiple restarts, since the bad cursor file survived restarts).

Deleting the cursor file and restarting fixed it: the fresh bootstrap completed in one batch (bodies_scanned=1 … has_more=false) and the cursor began persisting again.

Symptoms observed

Daemon (tracedecay daemon run) at ~90–110% CPU / ~1 GB RSS indefinitely; sample showed the tracedecay-store-* / tracedecay-native-* thread pools in continuous pread/stat I/O (not one hot loop on the main thread — main thread was parked).

~/.tracedecay/daemon.err.log repeating every few seconds, ~1,300+ recurrences:

event=retention_canonical_bodies roots_scanned=0 bodies_scanned=48 bodies_deleted=0 stored_bytes_deleted=0 bootstrap_complete=true has_more=true
event=retention_replay_metadata  rows_scanned=48 rows_compacted=0 rows_skipped=0 rows_skipped_oversized=0 metadata_bytes_removed=0 has_more=true

Note the contradiction: bootstrap_complete=true yet has_more=true forever, with zero rows deleted/compacted per pass and the same 48 scanned each time.

Stuck cursor on disk:
```
{"after_project_id":null,"after_unregistered_directory":null}
```
(~/.tracedecay/maintenance/retention-cold-store-cursor-v1.json) — never advanced across days.
A daemon restart alone did not help; each restart re-entered the same livelock.

Root-cause hypothesis

The persisted resume key and the store's actual contents drifted (rows pointed at by the cursor no longer match what the scan finds), so pagination cannot step past the batch: each pass reads the same 48 rows, the batch-finished/bookmark write is skipped or a no-op (0 deleted), and has_more keeps the loop armed. Same failure class as the hook-spool condition logged earlier in the same file:

hook_v2_replay_consumer: hook spool drain failed host="codex" spool=PendingWork cause=489 pending producer work redrive(s) stayed owed; the first was unavailable

i.e. a work queue whose head item is unavailable/unmatchable, so it can never drain. Two asks fall out of that:

Retention pagination should make forward progress (or bail with a diagnosable error) when a pass deletes/compacts 0 rows — infinite same-batch rescans should be impossible.
The cursor invariant (bootstrap_complete=true + persistent has_more=true + null bookmark) should be detected and self-healed, not silently retained.

Recovery that worked (for anyone hitting this)

tracedecay daemon stop
mv ~/.tracedecay/maintenance/retention-cold-store-cursor-v1.json{,.bak}
tracedecay daemon start

Post-recovery logs: bodies_scanned=1 … has_more=false, cursor recreated and advancing, CPU settles to single digits after the normal multi-minute reconciliation warm-up.

Environment

macOS 27.2 (26B123), arm64
daemon installed as launchd user service; ~953 MB user-sessions.db, ~186 MB global.db, 3 registered projects
~31 linked .t3 worktree project owners open at the time (amplifies per-project background load, but the livelock reproduced with the cursor as the single variable)

fix(retention): safely publish bounded cold-store cursors

Investigation at master e14cf673 and the current application integration lane:

The reported CPU and repeated 48-row retention batches are important, but the cold-store JSON cursor is not the frontier for either logged batch. cold_store.rs uses retention-cold-store-cursor-v1.json only for registered-orphan and unregistered-project directory sweeps; both cursor fields being null is an ordinary end-of-sweep state. Canonical-body GC instead uses transactional SQL canonical_body_gc_progress and canonical_body_gc_candidates (source); replay-metadata compaction uses the durable replay_metadata_compaction_store_id_v1 frontier (source). Both have a 48-row page cap.

Zero body deletions or replay compactions does not, by itself, prove that the same rows are being reread: GC discharges referenced candidates, and replay compaction advances the store-id frontier even for ineligible rows. We need consecutive transactional frontier/candidate-count observations, along with the actual maintenance-tick cadence and any failed/rolled-back writes, to distinguish a stalled loop from progress through a large pre-existing corpus. Do not reset or delete authoritative SQL state on this evidence.

A separate one-file cursor-integrity repair was safely staged on the existing held integration PR in 3a04a2c: randomized exclusive atomic staging prevents the old fixed .json.tmp symlink from truncating an unrelated file; cursor reads are bounded to 64 KiB and reject nonregular entries. It adds behavioral regression fixtures, but it is not a demonstrated fix for this CPU incident. Repository Bazel/native validation is pending. Keep this issue open until the real CPU cause is established and reproduced with a fix. Refs #4787.

fix(retention): expose bounded page progress

Partial #4787 progress is now staged on the single held application integration PR #4776, not master: c04a406 (Refs #4787).

The logged 48 scanned / 0 deleted and 48 scanned / 0 compacted counts previously hid whether pages advanced. Canonical-body GC now reports actual candidate discharge and verifies every scanned candidate was removed in the same guarded transaction, accounting for rows removed automatically by the body-deletion foreign-key cascade. Replay-metadata compaction reports the before/after transactional store-id frontier and refuses a nonadvancing nonempty page. Two new Rust behavioral tests cover >48 referenced canonical bodies and >48 ineligible replay rows: both must advance across distinct pages and quiesce without deleting live data.

Validation limits: The exact five-file commit and Git diff were reviewed. An independent SQLite foreign-key-cascade probe discharged pages of 48 and 4 with mixed deleted/referenced bodies; this is not repository Bazel validation. Rust/Bazel and native macOS tests were unavailable locally; new-head hosted CI has not passed.

This improves both diagnosis and fail-closed progress checks; it does not establish the cause of the reported sustained ~110% CPU incident. Compare successive new frontier_before/frontier_after and candidates_discharged logs against tick cadence and database size on the affected host before declaring the livelock resolved. Keep #4787 open. The PR includes owner-excluded Kev/decision-model work and must not merge to master, regardless of CI.

fix(retention): safely publish bounded cold-store cursors

fix(retention): expose bounded page progress

fix(maintenance): reject linked cold-store cursor parents

Partial #4787 filesystem-safety repair is now staged on the sole held application integration PR #4776, not on master: fcce64d (Refs #4787).

The earlier cold-store cursor hardening protected its .json.tmp staging file and destination, but still followed a symlinked maintenance/ parent directory. That could adopt a cursor from an unrelated directory or atomically replace that directory's cursor. The new path guard refuses a nonreal checkpoint parent on load and checks again before staging and rename. A Unix Rust behavioral regression proves that a linked maintenance parent cannot provide or publish a foreign cursor, leaves foreign bytes and entries untouched, then verifies a legitimate local checkpoint after removing the link. A separate Linux filesystem probe reproduced the earlier foreign-directory overwrite and the no-follow parent policy; it is not a repository Rust/Bazel test.

The exact single-file commit was independently diff-reviewed and published through an expected-old-head, force=false fast-forward; PR and remote branch now both show fcce64d. Exact-head hosted CI is in progress, not passed. Repository Bazel and native macOS tests were unavailable locally. This guard does not prove safety against concurrent directory replacement; the reported sustained CPU/root cause is also not established resolved. Keep #4787 open. #4776 remains strictly held from master merging because of excluded Kev/decision-model work.

fix(maintenance): anchor cold cursor access to directory handles

fix(maintenance): keep cursor capabilities portable

Partial filesystem race repair for #4787 is staged only on the held application PR #4776, not on master: d4098f4, followed by portability/style correction 4158e1b (Refs #4787). The prior metadata guard could be raced between checking maintenance/ and path-based opening/rename. Cursor reads now use a no-follow file open relative to a held no-follow directory capability; writes use exclusive no-follow staging and capability-relative atomic replacement, with directory durability sync. A Unix filesystem regression deliberately renames the open maintenance parent and substitutes a symlink to a foreign directory, verifying that neither read nor write adopts foreign cursor bytes. An independent native Linux filesystem probe passed (held-directory replacement, symlink refusal, staging collision). This is not a repository Bazel or native macOS test; exact-head hosted CI https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42332721883 was queued when checked. The 90–110% CPU incident is not proven resolved: the canonical-body/replay SQL frontiers are separate from the cold-store JSON cursor; sustained affected-host frontier/tick evidence is still required. Keep #4787 OPEN. #4776 must not merge into master because it contains excluded Kev/decision-model work.

fix(retention): advance cold-store cursor across filtered projects

Partial Refs #4787 — independently confirmed a distinct cold-store registry pagination stall and published a focused repair on the sole held application PR #4776, not master: 2f0f1ef.

orphan_stores::build_store_census_page pages registered project IDs (8 at a time) and supplies page.next_cursor from the project registry. Its entries are subsequently filtered to projects with profile_sharded store instances. run_cold_store_page incorrectly constructed its persisted after_project_id from the filtered store entries, falling back to the old cursor when a full registry page had no qualifying stores. Nine registered projects with no profile-sharded stores therefore produced a full first registry page with next_cursor = project-07 but empty census entries; the old code persisted null and replayed the first page indefinitely.

The one-file repair persists the registry's own page.next_cursor and deletes the incorrect inference helper. A new real Rust database-backed async regression seeds nine projects with zero profile-sharded stores and requires the first maintenance tick to persist project-07 and the second to finish with null, while preserving the normal non-store census behavior. An independent real SQLite pagination analog passed three nonzero assertions reproducing the old stall and new two-page termination. That is not a repository Bazel test or native macOS verification. Exact-head hosted CI run 42414761013 is in progress, not passed at this comment.

Reviewed the exact single-file Git diff and parent, published with expected_sha=0566171b..., force=false, and verified the PR and branch both at 2f0f1ef. Keep #4787 open: this removes one genuine cold-store progress defect, but the reported 48-row canonical-body GC and replay-metadata batches are driven by separate transactional SQL frontiers, so the affected-host ~90–110% CPU root cause remains unproven. The Kev/decision-model merge exclusion on #4776 remains absolute; no change landed on master.

Refs #4787 — new ownership/foreign-cursor write risk in the staged cold-store checkpoint capability, distinct from the already-fixed maintenance-leaf symlink race and from the original 48-body SQL livelock. Staged PR #4776 only; not on master.

At held application head 453a19b2e644b0bcdc15ff40739002e7849bf111, crates/tracedecay-maintenance/src/retention/cold_store.rs, checkpoint_directory constructs the ambient profile_root from the cursor pathname and opens it with Dir::open_ambient_dir(profile, ...). Only the maintenance leaf uses open_dir_nofollow. If a formerly owned profile-root pathname is replaced by a symlink to another real profile before capability acquisition, the ambient open adopts the foreign directory and the following no-follow maintenance open succeeds. persist_cursor_in_dir then publishes/replaces retention-cold-store-cursor-v1.json relative to that foreign handle. The existing held-directory regression swaps the maintenance path after opening and does not cover this ancestor/root substitution before opening.

Independent disposable Linux filesystem descriptor-relative analog (not TraceDecay Rust/Bazel/macOS/Windows execution): built owned/foreign profiles with separate maintenance/ and cursor sentinels; changed the live profile pathname into a symlink to the foreign profile; emulated open_ambient_dir / leaf O_NOFOLLOW / exclusive no-follow temporary / descriptor-relative rename. Five nonzero assertions passed: the ambient and leaf directory descriptors both identified the foreign profile, the foreign checkpoint was replaced, the original owned checkpoint remained unchanged, and no staging file appeared under the owned profile. This is an OS-operation reproduction, not evidence of an observed production incident.

Repair must bind checkpoint I/O to an admitted/trusted profile owner identity or held ancestor capability, not merely refuse symlinks on the final maintenance component or add a path precheck subject to TOCTOU. Preserve intentionally symlinked profile layouts explicitly; include a Rust parent/profile-path substitution regression and both symlink and real-directory replacement scenarios, then Bazel and native macOS/Windows verification. Coordinate the common ownership invariant with #4616 but keep fixes on the single held application PR #4776. No source change or merge made in this investigation. Do not close #4787: the affected-host 90–110% CPU canonical-body/replay-frontier incident is still unqualified. The Kev/decision-model merge exclusion remains absolute.

fix(retention): verify and expose transactional GC progress

test(retention): reopen read-cost fixture without denied PRAGMAs

fix(retention): verify canonical candidate discharge after triggers

style(retention): keep canonical GC test helpers rustfmt-clean

test(daemon): isolate failed-mount cleanup from graph readiness

chore(retention): reconcile shared test prerequisites

Refs #4787 — published the review-scope reconciliation, not a new CPU-resolution claim.

PR #4798 is now at af1045000305250ac1cefd94e87ea1ddeefc201c and shows exactly six retention-specific files above its explicit prerequisites. The existing shared scheduler correction is owned by #4821. The already-published failed-mount fixture correction is preserved byte-for-byte in new one-file PR #4827 (ee44f8c16b736c36075eaadfe4bee6551194951a). Landing order is #4821 → #4827 → #4798, retargeting each child to master only after its prerequisite lands; do not merge children into prerequisite branches.

The #4798 reconciliation is an exact-old-head nonforce fast-forward. Its full tree is identical to prior head 4984a54 (3f493dd3a148e21f7bbfe8f9f59a14392d00dd8e), so no peer update, retention fix, regression, fixture correction or assertion was reverted. #4813's writer-admission branch is untouched. Earlier reporting overstated the maintenance.rs overlap: its writer-admission hunks are separate from #4798's outcome-log hunks; filename overlap alone was not proof of a conflict. Formal COMMENT reviews were posted on #4798 and #4827, with ownership notes on #4813 and #4821.

Fresh exact-head Bazel CI is queued: #4798 https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42540490276 ; #4827 https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42540332272 . No local Bazel is available and no Cargo was invoked. These are not passing tests yet.

Keep this issue open: no affected-host post-fix CPU/frontier/timing run or master acceptance has been established. Protected #4776/Kev drafts, release #4763, production/infra, credentials and all schedules were not changed.

fix(retention): report committed canonical bootstrap frontiers

fix(retention): account for ineligible replay pages

Refs #4787 — published a new focused replay-retention diagnostics fix to PR #4798, commit 9a1c8a27d0 (2026-11-09 UTC).

Confirmed the observed rows_scanned=48 rows_compacted=0 rows_skipped=0 can be misleading despite actual durable frontier progress: the replay page counted only oversized or eligibility-passing-but-uncompactable rows as skipped, omitting ineligible retained representations entirely. The two-file fix counts every ineligible preserved row in rows_skipped. rows_skipped_oversized remains a subset; the compacted/preserved body data, byte budgets, DB transaction, and persisted store-ID frontier logic are unchanged. The database-backed Rust regressions now require 48+4 rows skipped with committed 0→48→52 progress and idle completion, five preserved representation kinds, and accurate counts across bounded large-ineligible JSON pages.

The source/test diff was independently inspected. The branch advanced by an exact expected-old-head, nonforce fast-forward, preserves other owners' work, and still changes only six retention files relative to its explicit prerequisite. New exact-head Bazel CI #42566057533 is pending; no new Rust test pass is claimed. Local Bazel is unavailable; Cargo was not run.

New separate dependency blocker: #4827 Linux fleet reported 182/183 passing Bazel targets and failed runtime_acceptance_suite::production_lsp_negotiates_and_projects_canonical_context with DeadlineExceeded while //crates/tracedecay:unit_test passed. I posted the exact failure on #4827; no timeout weakening or causal assumption. #4834's explicit Linux/Windows run and new #4798 CI are in progress.

Not incident closure: affected-host sustained CPU, consecutive committed SQL frontiers, tick cadence/duration and restart/quiescence evidence, and final master acceptance remain unproven. #4787 remains open, Refs not Fixes; Kev/decision-model and mixed #4776 remain DRAFT and excluded from merging, release #4763 read-only, other owner branches/infra/production/schedules unchanged.

fix(retention): scope SQL frontier logs to each store

Refs #4787 — published bfba4837 to the existing issue-specific PR #4798 via exact expected-old-head, nonforce fast-forward.

A further diagnostic ambiguity was confirmed: the daemon rotates among multiple mounted session databases, but its canonical-body and replay-metadata progress events previously omitted the database identity. Two different stores each processing 48 rows with a 0→48 SQL frontier could look like one store rereading the same page. The fix identifies the exact mounted store in both success and degraded events; it does not modify retention eligibility, transactional progress, scheduling or byte budgets. A new isolated subprocess regression requires four genuine operator-log events with identical 0→48 progress to remain uniquely attributable to two stores (and asserts nonzero test execution). Reviewed the exact one-file diff.

New exact-head CI #42574509920 is queued, not passing yet; local Bazel unavailable, no Cargo. This removes one reason log repetition could be misdiagnosed. The original 90–110% CPU incident is not proven resolved without affected-host CPU, consecutive same-store committed frontiers, tick timings/cadence and restart/quiescence evidence on master. Keep this issue OPEN; do not claim Fixes.

The separate prerequisites #4837 (DRAFT) → #4834 (DRAFT) → #4827 → #4798 still require their own reviews/CI. No prerequisite/other owner branch was changed, and #4776, Kev/decision-model drafts, release #4763, production, tags, infrastructure, credentials and schedules remain untouched.

fix(retention): refuse orphaned canonical bodies on vetoed delete

perf(retention): batch canonical bootstrap writes

perf(retention): batch observation bootstrap writer submissions

fix(retention): verify bootstrap seed integrity before advancing

Navigation Menu

Repository navigation

fix(daemon): retention maintenance livelocks on a stale cold-store cursor — same 48 bodies rescanned forever, daemon pinned at ~110% CPU #4787

Summary

Symptoms observed

Root-cause hypothesis

Recovery that worked (for anyone hitting this)

Environment

13 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

fix(daemon): retention maintenance livelocks on a stale cold-store cursor — same 48 bodies rescanned forever, daemon pinned at ~110% CPU #4787

Description

Summary

Symptoms observed

Root-cause hypothesis

Recovery that worked (for anyone hitting this)

Environment

Activity

ScriptedAlchemy commented on Nov 6, 2026

ScriptedAlchemy commented on Nov 7, 2026

ScriptedAlchemy commented on Nov 7, 2026

ScriptedAlchemy commented on Nov 7, 2026

ScriptedAlchemy commented on Nov 7, 2026

ScriptedAlchemy commented on Nov 7, 2026

13 remaining items

ScriptedAlchemy commented on Nov 9, 2026

ScriptedAlchemy commented on Nov 9, 2026

ScriptedAlchemy commented on Nov 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions