[ScriptedAlchemy]

Current Linux prerequisite

The previous Linux fleet stopped after both anonymous launcher API reads returned 403 and their upstream release URLs returned 504. The fixed shared download is included through the draft fixture stack at 56a0798d58. The two feature-file patches are unchanged. The existing acceptance and draft prerequisite holds remain.

Refs #4481. Keep the issue open for native macOS idle-CPU and 24-hour store-growth acceptance. HOLD MERGE pending independent review, CI, and the prerequisites below.

Relation receipt recovery

An idle scheduler pass reads the earliest eligible retry from the covering deadline index and returns that exact deadline. For due work, a short creation-order probe can finish a dense page immediately. When each eligible state has at most four due deadlines, equality seeks retrieve the first page plus one key from each state/deadline range. A bounded merge selects the global page in strict creation, session, generation order. With more fragmented deadlines, growing probes compare the creation-order distance to the next page against the size of the due population before choosing an index.

All probes share one read snapshot and one SQLite clock sample. Only returned page items hydrate journal data. The one-row sentinel carries keys, so its payload cannot consume the actor’s read budget. Scope mismatches, missing journals, cancellation, and retry wakeups retain their typed outcomes. The existing schema and indexes serve all paths. Owner review · Mixed backlog review

Verification

The production SQL ran with the bundled SQLite 3.53.2, QPSG enabled, 75,000 registered-schema receipts, real JSON journal rows, and a one-receipt page. Counts include the eligibility check, selection probes, and journal fetch. All ten tested distributions returned the original first identity and sentinel.

Distribution	Previous head ca06fd7, SQLite VM steps	7a1e6b1
All deferred	50	50
Only the newest receipt due	1,072	1,374
All due	1,059	226
64 older permanent, 10,000 due	280,967	1,457
Same, 997 distinct due deadlines	280,995	3,737
58,616 sleeping, 16,384 due at one deadline	588,117	1,553
Same, three due deadlines per eligible state	588,117	2,164
Same, 997 distinct due deadlines	588,117	470,344

The highly fragmented last case still reads and orders due keys on successive pages. Its median query cost was 2.69 ms across 21 warm runs on macOS. With these two index orders, widely scattered deadlines and a long ineligible prefix can still make one small page linear in the smaller backlog. In the few-deadline case, the query plan used covering index range seeks, rowid lookups for bounded candidates, and primary-key journal lookups for page items.

On native macOS the registered-store daemon convergence path processed 21 consecutive one-item pages for each 75,000-row distribution. Median/p95 were 1.12/1.47 ms with a single late due receipt, 1.04/1.33 ms with all due, 1.14/1.30 ms with the permanent prefix and 997 deadlines, 1.37/1.70 ms with the sleeping prefix and three deadlines, and 4.51/5.45 ms with the sleeping prefix and 997 deadlines. The mounted refresh worker settled the sparse receipt with a typed scope mismatch in 7.44 ms, including observation. Two individually valid 34-MiB journals exceed the actor’s 64-MiB combined read budget. The production one-item page processed the first and returned the second as its sentinel without materializing both payloads.

On 7a1e6b1139, the complete macOS session-runtime suite passed all 196 tests and the session-temporal-store suite passed 186 with one existing ignore. The maintained registered-store tests cover tied creation times across both eligible states and several real deadlines, page boundaries, scope refusal, missing/malformed journals, cancellation, and the exact future wakeup. The SQLite plan-stability checker and Rust formatting passed.

On a5ddc7e69f, the hydration join explicitly puts the bounded page before the receipt table. The bundled SQLite 3.53.2 registered store used composite primary-key seeks into receipts and journals for all three candidate paths, both with the previous SQL and with the explicit join order. On 75,001 journal-backed receipts the production page processed the lone due row behind 75,000 sleepers (no sentinel), the first all-due row (sentinel), and the earliest of 37 fragmented due rows (sentinel). System SQLite 3.51.0 also used these two key seeks before and after ANALYZE. The four maintained relation recovery, order, corruption, and cancellation tests and the SQL plan-stability checker passed. Hydration review

Stack

This PR targets #4833. The prerequisites in order from master are #4830, concurrent LCM publication, #4831, foreground projection during history, #4832, daemon readiness in the storage fixture, and #4833, the dashboard retained-fact fixture.

The Linux fleet on a5ddc7e69f passed all 183 targets, with 63 executed. The global DB unit suite ran in 243.3s and the store-runtime unit suite in 59.2s. The separate Clippy and Dashboard checks stopped during dependency fetch with HTTP 504.

The prior Linux fleet run passed all 89 dashboard API cases and 36 daemon cases and reported a scheduler supersession assertion and a native-host cold index deadline. The scheduler fixture correction passed locally. The cold index timeout did not reproduce across six native macOS mode/phase journeys or two independent Linux controls.

The current verification run checks b429528654 against #4833 at 8d8de435cb. This head includes the Linux launcher download prerequisite. The 2-file feature diff has the same paths and stable patch ID after updating the prerequisite.

[changeset-bot]

⚠️ No Changeset found

Latest commit: f306a33

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[ScriptedAlchemy]

Review/ownership blocker before merging this #4481 slice: the PR head advanced from f085c0f to 1430ba1 while this issue was being checked. The new commit is test(automation): bind scheduler fixtures to backend identities (Refs #4532), changing only crates/tracedecay/tests/automation_runner_test/jobs.rs. It addresses the separately failing automation-runner baseline, not this PR's relation-recovery source/tests, and introduces a third file into the previously two-file issue-specific diff. The previous Bazel attempt passed the affected session-runtime/session-temporal-store/LCM targets and Windows, but its exact-head evidence applies to f085c0f, not the new head. The new run must finish before current-head claims. The new fixture tests appear behaviorally motivated, but Refs #4532 names the separately owned graph-publication issue; confirm the proper owner and keep unrelated CI fixture repair independently reviewable rather than quietly mixing acceptance. No force pushes, merges, protection bypasses, or changes to the peer fix are authorized here. Issue #4481 remains open pending actual macOS daemon CPU and 24-hour store evidence.

[ScriptedAlchemy]

Published b0f6277 (Refs #4481): the history-interleaving test fixture had introduced OnceLock::wait() inside its async run_pass, blocking the very native worker required to poll live projection alongside in-flight history. The fixture now uses a durable async tokio::sync::watch release latch, preserving the release-before-poll guarantee without pinning the worker thread. The retained real scheduler supersession test still checks both named/aggregate joins. This fixes a test behavior defect, not an established production idle-CPU cause. Exact-head CI is pending; no Bazel or macOS daemon run is available locally. The PR's unrelated automation/dashboard/storage fixture changes still need independent ownership review; do not treat previous head checks as validation. #4481 remains open.

[ScriptedAlchemy]

New #4481 performance review blocker (current #4810 head 7ccb4247bd754e3be0255f0f81f3758f38a4b7ac): the idle fast path does not bound the mixed due/deferred pass.

The covering-index MIN(recovery_next_attempt_at) probe correctly skips the full recovery query when nothing is due. But as soon as one receipt becomes due, the original ordered candidate query still executes:

WHERE receipt.state = 'pending'
  AND receipt.recovery_state IN ('pending', 'retryable')
  AND receipt.recovery_next_attempt_at <= unixepoch()
ORDER BY receipt.created_at, receipt.session_id, receipt.generation
LIMIT ?1

SQLite chooses idx_session_relation_receipts_pending (state, created_at, session_id, generation) to preserve creation order; it may therefore walk all older sleeping retries to find one due receipt despite LIMIT 2. The new MIN probe does not constrain the second query.

Measured reproduction — query-level, not a daemon benchmark or Bazel test: SQLite 3.46.1, in-memory fixture with the repository's actual recovery-due/pending indexes, receipt/journal join and ordered query; 75,000 pending retryable receipts in created_at order, 74,999 with next attempt one day ahead and only the last-created receipt due. 70 warmed iterations: covering-index MIN probe median 0.0058 ms; original candidate query median 14.21 ms (p95 20.52 ms). SQLite's VM executed approximately 675,000 opcodes on the ordered query (measured using the progress handler at 100-opcode granularity).

Forcing idx_session_relation_receipts_recovery_due selects the one due receipt in 0.0117 ms (fewer than 100 VM opcodes in this fixture), but is not a general fix. With all 75,000 receipts due, the original creation-order index takes 0.0051 ms (about 100 opcodes), whereas the forced recovery-due index must sort all due rows and takes 41.63 ms (approximately 1,650,100 VM opcodes). EXPLAIN QUERY PLAN confirms the index and temporary sort difference.

Please preserve creation order and the one-sentinel pagination contract while bounding reads for both a sparse due receipt hidden behind deferred work and the all-due catch-up case. Add nonvacuous registered-store behavior and query-step evidence, followed by native daemon measurement. The existing 75k-all-deferred MIN benchmark is insufficient to establish mixed-queue idle CPU acceptance. No source commit or merge made from this review. Refs #4481.

[ScriptedAlchemy]

Refs #4481 — published two exact-head, non-force fast-forwards on the existing issue-specific PR: 586b094 and ca06fd7.

Root cause / source fix. The prior MIN(recovery_next_attempt_at) probe correctly skipped all-deferred queues, but the follow-on creation-ordered query still scanned all older deferred receipts whenever even one later receipt became due. Forcing the recovery-deadline index alone instead sorts the entire all-due backlog. The recovery page now examines a 64-row creation-order prefix; if it contains a full due page and sentinel, it retains the creation index. Otherwise it probes at most 16,384 indexed due entries, then chooses the deadline index for sparse work or the creation index for bulk catch-up. Both paths use the same SQLite snapshot deadline and preserve creation ordering and one-sentinel pagination.

Behavioral regression: the real registered-store test now adds 256 older FK-valid sleeping receipts before two existing due publication receipts. It checks first-page permanent settlement of the older poisoned receipt, second-page recovery, the future retry deadline, then wakes all 256 and checks that the dense creation-ordered path settles the oldest receipt and still advertises the next page. This extends the original malformed-journal/scheduler test; no mocked recovery result.

Independent query-level reproduction, not a daemon benchmark or Rust test: SQLite 3.46.1, warmed in-memory 75,000-row schema using the repository's two receipt indexes and the candidate/journal join; 31 iterations per layout, median end-to-end SQL-plan selection and row selection. One due receipt behind 74,999 deferred: 11.86 ms original → 0.029 ms revised. All due: 0.005 → 0.021 ms. A final 20% due: 8.37 → 3.54 ms. A final 40% due: 5.96 → 7.53 ms, a measured intermediate-distribution tradeoff, not grounds to claim universal speedup. All result identities matched the original query. This does not prove macOS daemon CPU or 24-hour store growth.

Exact-head Bazel CI 42554361971 is queued. No Rust tests were run locally because the checkout/Bazel toolchain are absent; no Cargo was used. Do not merge or close #4481 on this query-only evidence. Native idle-daemon CPU, retry-loop convergence and 24-hour bounded-store-growth acceptance are still outstanding; the stacked prerequisite PRs remain separate.

[ScriptedAlchemy]

Refs #4481 — exact-head CI infrastructure blocker on a5ddc7e69f9412db0ac9e28e942bbd117aed4a5b, not an established source/test failure.

In run 42560172386, Clippy job 128983850650 and Dashboard job 128983850753 failed during Bazel external-repository loading, before application analysis/tests could complete. Their logs independently show HTTP 504 Gateway Time-out fetching the pinned hermetic-launcher release asset runfiles-stub-x86_64-linux; Clippy also failed to fetch finalize-stub-x86_64-linux. Disk had ample capacity (89 GiB available on Clippy, 96 GiB on Dashboard), so this is not the prior disk-exhaustion failure.

I attempted targeted reruns of both failed jobs, but GitHub rejected each with HTTP 403: The workflow run containing this job is already running. The Linux/Windows jobs remain in the workflow queue. Retry only these failed jobs after the parent run is complete; do not change CI policy, pinned dependencies, or source to hide an external 504. Passing prior-head checks cannot validate this head. No merge or issue closure is warranted, and native macOS idle-CPU / 24-hour bounded-store acceptance remains outstanding.

[ScriptedAlchemy]

Refs #4481 — new recovery-query scaling blocker on current #4810 head 32a234b8e88479fd4cc41e3bb45de134a4987260. This is separate from the already-addressed one-due-row and <=4-deadline cases.

The current RECOVERY_DEADLINES_SQL enumerates only four due deadlines per eligible state. When a backlog has five or more distinct due deadlines, complete_bands is false. With a long older deferred prefix, page_in_prefix cannot find the page early, so recovery_candidate_query eventually selects idx_session_relation_receipts_recovery_due. Its ORDER BY created_at, session_id, generation LIMIT ?1 then sorts every due receipt again on each small recovery page; LIMIT only bounds the result, not the sort input. This can recreate the expensive per-page loop that the change aims to remove.

Reproduced using SQLite 3.46.1, the actual index shapes, a warmed in-memory database, 75,000 older sleeping receipts, and later due receipts spread over eight distinct deadlines (all in one eligible state). An exact copy of the current probe choices selected deadline-sort at windows 256 / 4,096 / 16,384 for 1k / 10k / 50k due receipts. The selected candidate-query EXPLAIN QUERY PLAN includes USE TEMP B-TREE FOR ORDER BY. Median of ten warmed candidate-only queries, requesting two keys:

Later due receipts	Current deadline-sort	Experimental eight-band-capable indexed query
1,000	0.117 ms	0.027 ms
10,000	1.087 ms	0.027 ms
50,000	6.398 ms	0.038 ms

The comparison query used the PR's indexed bands design with room for 16 distinct deadlines; it is not a committed fix. Simply increasing the four-deadline constant would move the cliff to 17 deadlines, so the review requirement is a bounded/amortized recovery strategy and a nonvacuous regression covering >4 fragmented due deadlines behind a large deferred prefix over successive daemon recovery pages. Keep ordering, sentinel pagination, snapshot clock, and cancellation guarantees. These are SQL query-level measurements, not daemon CPU or Bazel evidence; host acceptance remains outstanding.

The current head's Linux fleet is in progress and does not establish this scaling property. Please hold #4810's merge pending a real fix/behavioral evidence, alongside its stack prerequisites. No CI bypass or unrelated owner changes.

[ScriptedAlchemy]

Refs #4481. Published f306a33 via an exact-previous-head, non-force fast-forward. The new registered-store behavioral regression installs 512 older sleeping relation receipts and 24 newer receipts with 24 distinct due deadlines. It drives the real recovery method through 24 successive one-item pages, asserting each page settles exactly the oldest due receipt, maintains the sentinel until the final page, and leaves the sleeping prefix's next retry deadline intact. This exercises the fragmented-deadline fallback rather than only the four-band fast path. Only the existing #4810 issue-owned test hunk was added; the #4830–#4833 draft prerequisite stack and other owners' changes were preserved. Local Bazel is unavailable here, so the new test has not yet passed CI; do not treat it as verified. HOLD MERGE and keep #4481 open pending exact-head CI, independent review, native macOS idle-CPU/retry convergence, and 24-hour over-bound store growth evidence on master.

[ScriptedAlchemy]

Refs #4481 — new exact-head CI blocker on #4810 at f306a33a1be7967eacab8c4a1f31f4d22de0fff7 (2026-11-09). Linux fleet job 129187265674 completed with 182/183 Bazel targets passing; //crates/tracedecay:dashboard_api_test failed one of 89 tests (88 passed):

memory_curation::automatic_fact_receipt_endpoints_expose_terminal_applied_and_quarantined_receipts panicked in crates/tracedecay/tests/common/mod.rs:1587: "dashboard server did not become ready" after the fixture's 160 HTTP readiness attempts. The captured output nevertheless contains tracedecay dashboard listening on http://127.0.0.1:33643 and Dashboard bound to launch project. The aggregate Test Linux check also failed only because its fleet prerequisite failed. Disk was not exhausted (78 GiB available).

The changed #4810 files are exclusively crates/tracedecay-session-temporal-store/src/relation_projection.rs and crates/tracedecay-session-runtime/src/lcm_effects/tests.rs; neither changes the failing dashboard readiness helper or automatic-fact test. #4833 owns two other dashboard fixture changes in the draft prerequisite stack, but its patch does not modify this failing test's start_dashboard_fixture(false) call. This is therefore a new observed verification failure, not proof of a #4810 source regression nor an established baseline flake. On this exact head the session-runtime Bazel unit target executed and passed in 88.1 s, including the newly added 24-deadline behavioral regression; the temporal-store unit target passed from cache. This validates the test behavior, not the unresolved large-backlog scaling property or macOS daemon acceptance.

Run 42613918761 remains in progress with Windows running. Retry the failed fleet job only after the parent run finishes, then investigate readiness timing from the underlying fixture if it reproduces. Keep the 160-attempt readiness assertion and CI policy unchanged, and keep the #4810 merge hold: the separately documented highly fragmented query cost and native macOS daemon idle-CPU / 24-hour store-growth acceptance remain unproven on master. No merge, issue closure, protected-draft edit, or peer-owned change is justified.