[ScriptedAlchemy]

Refs #4756 — partial session re-import performance repair (verification pending)

On 3e6aa1f5cd, the workspace rustfmt gate and all 518 native global database tests passed, including the newly added rewrite-queue regression.

Not Fixes #4756. This PR removes measurable SQLite projection-queue amplification and adds restart/batch regressions. It does not yet prove durable mid-scan Codex continuation or the reporter's four-hour high-CPU and 125–160 ms profile write problem resolved.

On 5aa785428b all 515 native macOS global database tests passed, including retry deadlines that become due during recovery and the source-presence batching regression. The old test expected a transaction telemetry counter to measure individual SQL statements and failed on the real registered writer. A SQLite trigger now observes the executed statement boundaries on that writer. For 257 new records it measured pages of 128, 128 and 1. A second trigger rejects redundant updates on identical reoffers. The test also verifies that a malformed last page rolls back earlier pages and that the final offer for a duplicate identity wins. The final commit 89bd943319 also reran the full 515-test global database suite successfully, and both the repository-wide formatter and strict library/test Clippy targets passed.

On c3ddf7348c, the registered SQLite writer also measures every executed statement. Both 257 fresh offers and 257 identical reoffers execute exactly three page statements above the measured transaction baseline. Injecting an extra SELECT 1 into the reoffer transaction made the regression fail (5 statements versus 4); restoring the production code passed. The complete global database and SQLite runtime suites passed 515 and 426 tests respectively. Repository formatting and strict Clippy passed.

Root causes addressed

• N+1 queue scans: each projected item previously repeated a minimum-sequence queue-head probe in the same write window, repeatedly traversing a prefix of parked terminal collisions. The drain now selects one bounded, sequence-ordered window and retains per-item atomic rollback and head-of-line deferred-retry ordering.
• Idle SQLite writer contention: empty queues, permanently parked collisions, deferred head rows (even when later rows are due), and active rebuilds must not take a writer lease. A read-only sequence-head/rebuild probe returns accurate pending status and is reevaluated on the next poll; the writer transaction still independently validates work when a head is ready.
• Restart retry re-arm amplification: terminal collision and already-due rows no longer consume re-arm batch windows. A covering deadline index supplies the upper deadline/rowid key. Every read snapshot walks at most one page, then releases the WAL. Foreground retries stamp their database mount in the same transaction as their deadline. The writer rechecks mount, rowid, observation ID, attempt count and deadline before rearming a row.
• Stable index plans: sequence-head reads use a fixed literal terminal sentinel so SQLite can reliably select idx_projection_queue_active_sequence under its query-planner stability guarantee. The retry-deadline index and historical-data index installer permit nonterminal probes without scanning the terminal prefix. Repository-level next-head reads and projection-frontier repair use the same sentinel predicate.

Behavioral regression evidence

• Real persistent profile DB: five observation records, a parked terminal collision and a retry-deferred sequence fence; a fresh runtime re-arms the previous mount’s deferred row, resumes from checkpoint 3 at the exact next observation, and never duplicates earlier messages. A newly deferred head still fences later ready observations while this mount remains active.
• A separate SQLite BEGIN IMMEDIATE reservation verifies that empty, terminal-only, deferred-head and rebuild-blocked projection polls do not acquire a competing writer; releasing the fence or admitting a new observation resumes durable progress.
• Registered global DB retry queue: interleaved terminal collisions, due and deferred rows; asserts exact modified rows, one settled writer lease per nonempty batch, zero writer leases on idle re-arm, and continuation after a true DB reopen.
• Independent offline SQLite selection checks matched old query ordering and partial-index selection. A synthetic 30,000-parked/1,000-ready workload measured ~59 ms for repeated per-item head probes versus ~2.4 ms for a single-window selection (~25x query-only difference). This is not an end-to-end TraceDecay benchmark or proof that the reporter had that distribution.

Verification and remaining acceptance

At b7cd1afd02, the original projection queue definition is byte-identical to master. Retry ownership uses a separate table whose rows follow the queue entry by foreign key. The deadline and its owning mount commit in one writer transaction. A saved queue from the shipped schema reopens and resumes without a reset or queue alteration. A queue missing its required retry counter is refused without modification. Injecting an ownership-write failure rolled the scheduling transaction back and preserved the previous retry. The full registered database suite passed 513 cases on macOS and the session suite passed 37 projection integration cases. Both affected Clippy targets, Rust formatting, and the query plan checker for 16 partial indexes passed. Hosted Linux run · Schema review

At 09a27629fe, the complete registered global database suite passed 511 cases on macOS, and 37 observation-projection integration cases passed. The tests exercised opening a database saved with the prior queue schema, preserving a live retry across recovery pages and physical rowid reuse, a successful WAL restart checkpoint between pages, and resuming an interrupted recovery. The SQLite plan checker passed for all 16 partial indexes, along with its six tests, Rust formatting and affected Clippy targets. The latest source stages at most a single page in memory and writes no candidate file. It supports queues that exceeded the former whole-backlog staging limit. Review discussion

At 84bf09170c8783507f46bd7b137b64a50dc8ad53, the complete //crates/tracedecay-global-db:unit_test suite passed all 504 tests on macOS in 58.01 seconds. All 37 //crates/tracedecay:session_suite observation projection integration cases passed, including the 10 queue and restart cases. The repository SQLite plan checker passed for all 16 partial indexes, and its six self-tests passed. Exact-head hosted CI.

With QPSG enabled on SQLite 3.54.1, a four-row update of a dense 100,000-row retry queue fell from approximately 700,100 VM instructions to 200. The empty read-side probe used the covering deadline index on an analyzed 100,000-row expired queue. The native writer regression expands 32 deferred observations to 256 under a fixed four-row batch. It fails on the earlier implementation (3,160 to 75,456 steps) and passes after the change.

Codex’s process-retained partial discovery position still does not survive daemon restart. Per-source admission amplification, the reporter’s repeated-restart profile, and measured end-to-end CPU and write latency remain open under #4756. This PR is ready for review to enable Linux CI. Keep the existing merge hold pending substantive review and hosted checks. Keep its queue changes separate from held mixed #4776, decision-model drafts and release #4763.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0099fe6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Follow-up at d265e31 — Refs #4756, partial restart write-amplification reduction.

Root cause: the background store-mount retry re-arm paginated every queue row with a positive deadline, including terminal collision rows whose deadline is i64::MAX. The UPDATE intentionally left those terminal rows untouched, but each such page still incurred its own writer transaction and commit on every restart. The selection now includes only genuinely retryable deferred rows, so parked collisions neither count against the fixed re-arm batch size nor cause empty write batches.

The new behavioral regression seeds 12 persisted queue entries (six terminal, three deferred, three already due), asserts exactly three re-arms in two write batches of size two, verifies that terminal rows and error history remain unchanged, and checks that another fresh re-arm finds zero write batches. An independent SQLite execution of the same selection/update showed old = 5 batches / 3 updated rows, new = 2 batches / 3 updated rows, and repeat = 0 batches / 0 rows. This is SQL-level corroboration only, not a Bazel Rust test pass.

No checkout or Bazel executable is available in this execution environment, so the Rust regression remains uncompiled here; exact-head CI was queued at review time. This does not establish resolution of the reported 125–160 ms projection writes or four-hour CPU saturation; no macOS end-to-end restart profile is claimed. Keep this PR draft and #4756 open pending that evidence. No changes to #4776, release #4763, other owner branches, or CI policy.

[ScriptedAlchemy]

Published 5466c909 as a narrow additional Refs #4756 improvement (not full issue acceptance).

Remaining hot path: after a projection becomes a terminal collision, its queue row remains parked at i64::MAX. The previous empty-queue fast path only checked whether any row existed, so every later terminal-only drain still opened and committed an otherwise idle writer transaction, reread the parked prefix, and could contend with foreground writes.

Fix: the pre-writer probe now checks for nonterminal work and uses a separately installed performance index on next_retry_at_micros. The actual write transaction remains authoritative when fresh work races the probe. The existing restart regression now holds an unrelated SQLite BEGIN IMMEDIATE writer while draining a queue containing only a persisted terminal collision: the drain must finish without acquiring that lease. After releasing the writer, the same reopened profile admits a sixth observation and must advance the durable checkpoint from 5 to 6, proving the skip is not cached.

Independent checks: SQLite WAL reproduction confirmed that a terminal-only probe returns no work while another connection holds a writer, the new covering index is selected by SQLite, and a subsequently inserted ready row is visible. Synthetic 30K-parked-row query-plan checks showed a covering-index probe rather than a full queue scan; these are not end-to-end TraceDecay CPU measurements.

Only observation_projection/rebuild.rs, observation_projection/schema.rs, and the existing session_suite/observation_projection/queue.rs regression changed. The commit is a non-force fast-forward from the independently checked old head 35da211f, with an exact expected-head lease. Hosted Bazel CI for this head is queued. No Bazel executable/checkout is present in this session; Rust compilation and nonvacuous tests are not yet verified. Do not merge pending that evidence and review.

This does not prove the reported 125–160 ms historical writes, multi-hour 1.7-core CPU usage, or durable Codex discovery continuation across daemon restarts are resolved. Leave #4756 open and this PR draft. The overlapping #4776 draft, decision-model work, and release #4763 were untouched.

[ScriptedAlchemy]

Follow-up at 1c08ac0c — Refs #4756, partial bounded-write improvement.

Root cause: even after the terminal-only fast path, a queue containing no due observations but one or more deferred retries still entered a SQLite writer transaction on every projection poll. A terminal collision plus deferred nonterminal row is sufficient to reproduce the needless writer lease; repeated polls can contend with foreground session writes without advancing the checkpoint.

Fix: query the earliest nonterminal deadline through the PR's existing next_retry_at_micros index before acquiring the writer. No nonterminal rows returns an empty settled batch; all-deferred rows return an empty pending batch (has_more=true) without a writer; any due row still enters the unchanged transactional, sequence-ordered projection path. No new index, schema shape, or public API was added by this commit.

Falsifiable regression: reopen a real profile SQLite store containing a persisted terminal collision and a deferred observation; hold an unrelated BEGIN IMMEDIATE writer reservation and require two projection polls to finish without waiting, while still reporting pending coverage and leaving the durable checkpoint unchanged. Release the writer, make the retry due, and require exactly one projected message and checkpoint advancement. The old pre-writer SELECT 1 path acquires the writer and fails this test.

Independent check: an isolated SQLite WAL rehearsal confirmed the read-only earliest-deadline query works while another writer holds a reservation, uses the covering retry-deadline index, and sees a subsequently eligible row. This is SQL-level evidence, not Rust/Bazel or host CPU verification.

This is a two-file diff, published via a single-parent, exact expected-old-head, non-force fast-forward from 5466c909. Exact-head Bazel CI is queued; no Rust test execution can be claimed yet because Bazel is unavailable locally. The 4-hour CPU/125–160 ms write incident and durable cross-restart Codex discovery continuation remain unproven, so keep this PR draft and #4756 open. No changes to mixed #4776, Kev/decision-model work, release #4763, CI gates, or other owners' branches.

[ScriptedAlchemy]

Published 6f32ff2f as a focused Refs #4756 follow-up, not full incident acceptance.

Root cause: the previous read-only projection probe used the minimum retry deadline over all nonterminal rows, not the first nonterminal observation sequence. If sequence 2 was deferred but sequence 3 was already due, each idle poll still took a SQLite writer lease and committed an empty projection window despite sequence 2 fencing the later row. Many terminal parked collisions also imposed repeated ordered-prefix work.

Fix: probe the lowest nonterminal sequence and compare its deadline. Use an indexed partial active-sequence lookup excluding permanent i64::MAX collisions both for the probe and for the bounded transactional queue window. When no head is ready, report pending coverage without a writer; when it becomes ready, keep the existing sequence-ordered transactional authority.

Regression: extended the real profile-store restart test with a third, already-due observation behind an earlier deferred one. Two polls while an unrelated connection holds BEGIN IMMEDIATE must complete without acquiring the writer. After releasing the reservation and rearming the deferred head, one-item windows must advance the durable checkpoint from 1→2→3 without jumping forward or duplicating message rows. This would fail on the preceding 1c08ac0c head.

An independent SQLite rehearsal with 30,000 parked rows confirmed the old earliest-deadline probe returned the later due row, the sequence-head probe returned the deferred row, and EXPLAIN QUERY PLAN selected the covering partial sequence index. The synthetic query-only timing is not an end-to-end CPU or writer-latency result.

This single-parent commit was applied by exact expected-old-head, non-force fast-forward, changing only observation_projection/rebuild.rs, observation_projection/schema.rs, and session_suite/observation_projection/queue.rs. Exact-head hosted CI is queued; Bazel is not installed in this runtime, so no nonvacuous Rust test pass or macOS restart profile can yet be claimed. #4756 and this draft remain open. Mixed draft #4776, other owners' PRs, Kev/decision-model work, release #4763, and CI infrastructure were untouched.

[ScriptedAlchemy]

Follow-up 71195bb7 — Refs #4756, partial write-amplification reduction only.

Root cause and fix: When the canonical observation_projection_rebuilds fence was active, an otherwise ready queued observation passed the earlier read-only deadline probe. The later transaction correctly suppressed ordinary projection, but every poll still acquired and committed an idle SQLite writer. The preflight now checks the same projector's durable rebuild fence alongside the sequence-first head and returns pending work without a writer lease; the in-transaction rebuild check remains authoritative if state changes after the probe.

Behavioral regression: active_rebuild_keeps_projection_pending_without_acquiring_writer admits a real observation, inserts the durable rebuild fence, holds a separate SQLite BEGIN IMMEDIATE writer, and requires two drains to finish as pending without acquiring that writer. After clearing the fence, a one-item drain must project the original observation and advance its durable checkpoint. The test would block/fail without this fix.

Verification: Independently reviewed the exact two-file commit diff. A separate SQLite WAL reproduction confirmed the active-fence probe result, partial sequence-index plan and writer-lock behavior, plus immediate visibility after fence removal. This is SQL corroboration, not a Rust/Bazel pass or macOS CPU profile. Hosted CI on this exact head is queued; local Bazel/checkout are unavailable. No Cargo used.

Keep this PR draft, #4756 open, and do not claim resolution of the reporter's 4-hour CPU saturation, 125–160 ms writes, or durable Codex in-progress discovery across restarts. Peer issue branches, held #4776/Kev decision-model work, release #4763, CI policy, and infrastructure were untouched.

[ScriptedAlchemy]

Follow-up 348d92f8 — Refs #4756, CI repair only (not issue acceptance).

The exact preceding-head Bazel workflow 42452435228 failed its rustfmt_check_2024 repository gates. Retrieved the complete hosted job logs and applied all eight emitted formatting hunks to only the two projection regression test files (observation_projection/rearm_tests.rs and session_suite/observation_projection/queue.rs). The new commit is exactly one nonforce fast-forward from 71195bb7, independently checked with GitHub compare (two files, no behavior changes). No tests were weakened or removed. New exact-head Bazel CI is queued; compilation, nonvacuous Rust execution, and incident-scale macOS CPU/write verification remain unproven.

PR remains draft and #4756 remains open. No modifications to held #4776, Kev/decision-model PRs, release #4763, other owner branches, or infrastructure.

[ScriptedAlchemy]

Follow-up source/CI repair at f7e4a6e7 and formatting-only aadc35a4 — Refs #4756, not full incident acceptance.

Exact failing gate: hosted Bazel run 42456607785 passed rustfmt but failed Check SQL plans stay index-backed under QPSG: eight bound comparisons to the terminal-retry predicate of the new partial idx_projection_queue_active_sequence index. Under SQLite's stable-plan mode, next_retry_at_micros < ? cannot imply that index's fixed i64::MAX predicate. This can silently regress into expensive scans despite the index.

Fix: Name the terminal sentinel as the identical SQL integer literal in all eight queries, adjusting the bind positions without changing semantic retry limits. The typed NextQueuedProjection reader now loads the first nonterminal sequence through the partial index and applies the current clock deadline after reading: a later due item cannot jump past an earlier deferred head. Extended the real reopen/retry regression to check both states via the public queue reader.

Review: GitHub compare verifies an exact-head, nonforce single-parent fast-forward from 348d92f8 to f7e4a6e7 (five changed source/test files); the follow-up is import formatting only. An isolated SQLite equivalence check compared the previous and revised head-selection results across 2,000 state/time combinations and checked index selection. These are SQL-level corroboration, not nonvacuous Rust/Bazel results.

New exact-head CI 42462245774 is queued. Do not merge: PR stays draft, issue #4756 stays open until Rust tests, durable mid-scan Codex restart continuation, and incident-scale CPU/writer performance are demonstrated. No change to mixed draft #4776, Kev/decision-model PRs, or release #4763.

[ScriptedAlchemy]

Exact-head verification blocker — Refs #4756

The 2026-11-08 CI run #42468061476 is success at head c1d1f67b64064d2eac6235e0bd398450843093a3, but that is not a Rust test pass. Repository gates (including rustfmt and SQLite QPSG checks) and Dashboard passed; Test Linux, Test macOS, Test Windows, and Clippy were skipped. The cause is the deliberate .github/workflows/ci.yml condition !github.event.pull_request.draft for the Rust jobs (including lines 426–428 and 787–790), not an absence of changed Rust paths. A targeted rerun of the skipped Test Linux job was accepted by the Actions endpoint but completed skipped again on attempt 2; rerunning cannot bypass the draft guard.

Keep this PR as the issue-specific partial Refs #4756 lane and #4756 open. Its real-Rust projection/reopen regressions have not executed, so no merge or complete-fix assertion should rely on the green aggregate CI badge. Verification requires a controlled, explicit Bazel Rust test run on the exact head (e.g. //crates/tracedecay-global-db:unit_test, //crates/tracedecay:session_suite, plus applicable broader targets), or a deliberate review-state decision with protection against premature merging. No workflow policy was weakened, and the held #4776 / Kev / release lanes are untouched.

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[ScriptedAlchemy]

Refs #4756 — published 50d28225 on existing issue-specific PR #4793 with expected-old-head 6aeabb84, non-force fast-forward.

Additional source-rewrite write-amplification cause: a source-presence row may remain after its observations row is absent (the registered-store >10,000-row fixture explicitly exercises this). The existing completion query included every such ghost because it lacked a projection disposition. retire_source_record then executed an empty projection-queue UPDATE and an empty receipt SELECT for each ghost, all within one bounded-policy SQLite writer job, before deleting the stale presence entries. A single 1,000-row SELECT page bounded materialization, not writer transaction length.

The fix selects an unoffered row for retirement only if it has a materialized observation without a disposition or an accepted queued revival requiring cancellation. A queued revival remains selected even with a missing observation. Already-disposed rows stay excluded. The authoritative final DELETE still settles all stale source-presence rows, and the entire completion keeps its existing atomic rollback semantics.

The previously registered rewrite_completion_retires_unoffered_records_beyond_the_query_cap test now also measures actual SQLite writer statement count over 10,003 persisted presence-only records, requiring sublinear writer commands, while retaining all original cleanup assertions. Separate registered tests still cover real projected retirements, rollback on provenance conflict and accepted revival. No tests/CI policy removed.

Independent SQL mechanism reproduction (SQLite, exact candidate predicates with primary-key indexes): 10,003 presence-only records produced 20,022 SQLite statements in the old per-record completion loop versus 6 after skipping no-op candidates; both deleted every stale presence row and passed PRAGMA integrity_check. A six-case truth-table check confirmed unprojected materialized observations and orphaned accepted revivals remain candidates; fully settled observations and presence-only ghosts do not. This is not an end-to-end TraceDecay benchmark or a native Rust pass.

New remaining bounded-write risk for review: if a rewritten source actually has many materialized, undisposed observations, complete_observation_source_rewrite still runs all their retire_source_record calls under a single transact_or_rollback writer job. The 1,000-row keyset page bounds SELECT result size but not total writer statements or the fixed 120-second job lease. Truly bounded write latency for such a source requires a separately designed, durable, interruption-safe retirement protocol without publishing partially retired source state; this commit deliberately avoids changing atomicity or adding a second authority.

Exact-head hosted Bazel run #42618257339 is queued. No local Bazel checkout/executable was provided; no Cargo and no claims of a new Rust test pass. Keep PR review/merge hold and #4756 open: cross-restart Codex mid-scan durability and the incident-scale CPU/125–160 ms write-latency acceptance are not yet proven on master. Protected #4776 / Kev drafts, release #4763, other owner branches and schedules untouched.

[ScriptedAlchemy]

Refs #4756 — published 71256708 to this existing issue-specific PR with expected old head 50d28225, force=false. Only registered_source_rewrite.rs and its registered-store behavioral test changed.

Root cause: source-rewrite completion used paginated SELECTs but retired all eligible observations and deleted all old source-presence records in a single SQLite writer transaction. A large real source could therefore exhaust the writer's fixed lease; a late provenance error discarded hours of completed work, causing the next run to replay every retirement.

Fix: consume all old-generation source-presence rows in 128-record writer windows (including ghosts and already disposed records); run retirement only where there is a live, unsettled observation or an accepted revival; delete exactly the processed old-generation key range in the same transaction; commit before fetching another page. Source presence plus the existing pending-rewrite row now form the durable restart frontier, without a new table, temporary staging file or pinned WAL snapshot. A failed page rolls back, while earlier committed pages survive for retry/restart. The final page removes the pending rewrite marker atomically. The source-generation guard is rechecked in every transaction.

Nonvacuous registered-store regressions: 10,003 presence-only records must complete in exactly 79 committed <=128-record pages and without per-record redundant writer SQL; the real 1,005-projected-record fixture injects a provenance collision in the final page, requires previously committed dispositions/provenance cleanup to remain durable, closes/reopens the registered profile DB, and verifies the next completion retires only the remaining observations, preserving the reoffered survivor and final cleanup. The prior whole-source rollback assertion was updated to the new per-page atomicity contract; the failure and final-state assertions remain.

Independent SQLite 3.46.1 reproduction (not a Rust test): indexed page query and delete matched the production SQL. A 1,005-row workload failed on the last page after 7 committed pages / 896 dispositions; reopening the DB retained the pending rewrite and frontier, then retired the remaining 108, yielding 1,004 dispositions, one retained survivor, zero pending rewrites and PRAGMA integrity_check=ok. EXPLAIN QUERY PLAN used the source-presence primary key and correlated PK lookups, not a full table scan.

Verification status: exact-head Bazel CI #42621820543 is queued; Bazel and the Rust toolchain are unavailable in the local execution environment, so no passing Rust test result is claimed. This is a focused, reviewable partial fix, not Fixes #4756. Durable Codex mid-directory restart continuation and reporter-comparable 4-hour CPU / 125–160 ms writer acceptance remain unproven. Leave #4756 open and PR #4793 unmerged until reviewed/tested. Protected drafts, #4776, release #4763 and other owners' branches were not modified.

[ScriptedAlchemy]

Refs #4756 — published 6aab5111 on existing issue-specific PR #4793 by expected-old-head 71256708, non-force fast-forward.

Actual CI blocker cleared in source: exact-head run #42621820543 failed //crates/tracedecay-global-db:rustfmt_check_2024 in repository gates. Its full job log identified three formatting hunks only in src/tests/source_rewrite.rs (two broken committed_transactions chains and one multiline assertion). Applied precisely the formatter's replacement lines; no test conditions, production behavior, budgets or CI policy changed. All 8 other repository-gate test targets passed in that run, but that run does not establish the Rust regressions passed.

New exact-head hosted Bazel run #42635755194 is queued; require fresh nonvacuous global DB tests plus CI and review before merge. No Bazel runner is installed in this authoring environment; no Cargo used. Durable Codex mid-directory restart continuation and measured four-hour CPU/profile writer-latency acceptance remain unproven. This is not Fixes #4756; keep the issue open and the PR under merge hold. No other branches, protected drafts, release #4763 or schedules were changed.

[ScriptedAlchemy]

Refs #4756 — source-rewrite survivor-prefix keyset repair, published on this PR at 6a0abf5558db32b50866b6e15e5463d5c2ed30c5.

Independent review found that the new 128-record retirement pages were still selecting and deleting from the beginning of the (source_key, observation_id) primary-key range on every page. When thousands of already reoffered, current-generation records sort before an obsolete suffix, each page repeats the unchanged prefix scan, so total SQLite work grows with pages × survivors.

• 57ff9671: pass an in-memory lower keyset bound through both selection and deletion, while retaining durable delete-per-page restart progress and rechecking the whole source before clearing a pending rewrite if a concurrent older offer could have arrived behind the keyset.
• a169d54d: correct the superseded-rewrite rollback tuple to match the expanded three-field return (caught on source review).
• 6a0abf55: skip the redundant full-source recheck when the first and only page already covered the entire source under one writer lease.

New registered-store Rust regression measures the actual writer's SQLite VM steps, with 4,096 surviving records preceding 127 versus 1,023 obsolete records. It also checks exactly one versus eight committed retirement pages and verifies all survivors remain while all older presence rows disappear. The growth threshold fails the original prefix-rescanning algorithm in an independent SQLite approximation.

Independent SQLite reproduction (not a TraceDecay end-to-end run): for the same primary-key layout, empty observation/queue tables, and the equivalent query shape, 1,023 older records cost 311,510 → 98,580 approximate VM steps over eight pages (~3.16× less); 127 older records cost 49,682 → 49,696 in one page, with eight/one page counts unchanged and zero older records left. Instrumentation used SQLite's progress callback at every virtual-machine operation; these are model results, not production latency measurements.

The exact-head Bazel workflow is queued; no new Rust test pass is claimed, and local Bazel is unavailable. The PR remains unmerged. This is partial only: Codex mid-directory restart position and the reporter's four-hour CPU / 125–160 ms end-to-end write-latency acceptance are still open. No protected draft, release, another owner's branch, CI policy, or schedule was modified.

[ScriptedAlchemy]

Refs #4756 — published 54e8f807 onto the existing issue-specific PR #4793 using expected-old-head 6a0abf55 and force=false. This is a test/CI repair, not incident acceptance or Fixes #4756.

Inspected the latest exact-head Bazel run #42643202619, including both failed jobs:

• Clippy failed on two real warnings in the new registered-store regression: clippy::drop_non_drop at source_rewrite.rs:276 and clippy::needless_borrow at :608. Scope the first writer handle in a lexical block and return its already-retired count before reopening the runtime; remove the redundant borrow. The failure assertions and restart continuation checks are unchanged.
• Linux timed out in //crates/tracedecay-global-db:unit_test with the >10,000-row ghost-presence regression still running. Its fixture seeded 10,003 rows with 10,003 serial SQLite execute calls; the regression now seeds exactly the same identities with 79 recursive-CTE statements of at most 128 rows, and directly asserts all 10,003 rows exist before exercising production. An independent SQLite :memory: reproduction verified 10,003 distinct IDs with exact min/max and offsets (SQL-only, not a Rust pass).
• Separate remaining blocker: //crates/tracedecay:transport_acceptance_suite failed 2 of 30 tests at stale_sessions_store_reset.rs:426: code callers returned {\"problem\":\"reset_required\"} rather than evidence over refused session stores. This needs a source-backed isolation diagnosis before merging. No assertion, CI rule, or timeout was weakened.

Only crates/tracedecay-global-db/src/tests/source_rewrite.rs changed; current master, other owners' branches, protected drafts (including #4776 and Kev), and release #4763 were untouched. The exact new-head Bazel CI #42660546473 is queued; no new Rust pass is claimed. Keep PR and issue open until full acceptance (including durable Codex mid-scan restart and incident-scale CPU/write latency) is demonstrated.

[ScriptedAlchemy]

Refs #4756 — published 656ae03b onto this existing issue-specific PR by exact-old-head 54e8f807, force=false. Only registered_source_rewrite.rs and its registered-store test changed; no other branch, protected draft, release, or CI policy was modified.

New repeated-restart writer-amplification mechanism: begin_observation_source_rewrite reads an unfinished generation and includes it in generation IN (?4, ?5). When an interrupted source re-import invokes begin again for that same generation, ?5 == ?2. The old source-presence UPDATE and accepted-revival queue UPDATE therefore physically update every already-advanced retained-prefix row to its existing generation, under a writer lease. This cost recurs on daemon restart even though no new source observations are present.

Focused source fix: both indexed UPDATEs now exclude entries already on the target generation (generation <> ?2 / presence.generation <> ?2). Old-to-new and genuinely newer generation transitions still update exactly their eligible rows. No schema, frontier, or deadline changes.

Falsifiable registered-store Rust regression: repeated_source_rewrite_begin_preserves_already_advanced_prefix persists two real projected observations and a queued accepted revival, advances a retained prefix from generation 1 to 2, installs SQLite BEFORE UPDATE triggers that abort on an unchanged presence or revival generation, and re-enters the same source rewrite. The prior SQL would fail the no-op triggers; the guarded SQL must succeed and preserve both authorities. A subsequent genuine generation 2→3 transition must still advance both presence and revival. This tests the production writer on a registered profile store, not a source-shape grep.

An independent SQLite statement/trigger reproduction verified the old same-generation UPDATE fails and the guarded version succeeds; that is SQL-level mechanism evidence, not an executed Rust/Bazel regression or incident-scale benchmark. The new exact-head hosted Bazel run has not yet established a passing Rust result. The preceding run #42660546473 was still running when inspected. Retain the review/merge hold; #4756 remains open.

Still missing for full acceptance: durable Codex mid-directory restart continuation (the current ReadDir position remains process-held), per-provider eligible-file/admission progress across restarts, and incident-scale before/after measurement of the reported four-hour CPU and 125–160 ms SQLite writes. No Fixes #4756 claim.

[ScriptedAlchemy]

Refs #4756 — new exact-head CI execution blocker, not a code/test failure diagnosis.

Reviewed current master d6acbc49, root AGENTS.md (recursive tree contains no nested AGENTS.md), issue #4756 and its paginated discussion, #4793's current head/reviews/changed files, and protected draft #4776. The existing issue-specific PR head remains 656ae03b6db8516553fa8d5dc8a3bc11aa6b089f; #4848 is already merged separately.

Hosted Bazel run #42669877759, attempt 1 finished failure even though Repository gates and Feature gates passed: the Test Linux aggregate job failed solely at Require every Linux test shard, reporting Linux test shards finished with result 'abandoned'. The available aggregate log has no individual failing Rust assertion, so this attempt does not verify or refute the newest registered-store source-rewrite regression. Requested re-run of failed jobs only; attempt 2 was queued at latest inspection. No CI configuration or policy was weakened.

The previous complete Linux fleet run #42643202619 had a timeout in the pre-optimization >10k-row fixture and two transport acceptance failures; the subsequent fixture repair and no-op generation update still require exact-head, nonvacuous Bazel confirmation. The transport failures should not be attributed to #4756 without isolating them against master.

Remain blocked from merge/closure: passing exact-head Rust/global-DB tests and independent source review, durable Codex mid-directory restart continuation, plus incident-scale provider-attributed CPU and 125–160 ms writer latency before/after measurements. Existing PR stays open, Refs #4756 only; no new production changes or branch ref moves this run. Protected drafts, release #4763, and other owners' work untouched.

[ScriptedAlchemy]

Refs #4756 — partial repeated-restart writer-amplification fix published at 9573a033 on this existing issue-specific PR.

Root cause: begin_observation_source_rewrite already skipped identical retained-presence and accepted-revival UPDATEs, but its observation_source_rewrites INSERT ... ON CONFLICT DO UPDATE still updated the pending rewrite marker on every identical restart begin, even when generation and retained_through had not changed. That is an unnecessary write and trigger dispatch under the profile SQLite writer lease.

Source: make the conflict update conditional on a genuine generation or retained-offset change. The real pending rewrite marker is still inserted on first use, and new generation/retained-offset transitions still update it. No schema or budget changes.

Regression: extended the existing registered-store repeated_source_rewrite_begin_preserves_already_advanced_prefix to add a SQLite BEFORE UPDATE trigger that aborts unchanged rewrite-marker updates, alongside the existing presence/revival triggers. A new assertion proves a subsequent genuine generation transition advances the pending rewrite marker. This is nonvacuous: independent SQLite reproduction of the original unconditional upsert fails under the trigger; the conditional upsert passes, and genuine retained-offset/generation changes remain possible. That SQLite exercise is not a Rust/Bazel test pass.

Published with exact expected old head 656ae03b6db8516553fa8d5dc8a3bc11aa6b089f, force=false; only registered_source_rewrite.rs and tests/source_rewrite.rs changed. New exact-head Bazel CI #42696633132 is pending. No Rust test success or incident-scale performance acceptance is claimed. Keep #4756 open: durable mid-directory Codex restart continuation, provider-attributed CPU and 125–160 ms profile write latency remain unproven. Protected draft #4776, decision-model drafts, release #4763, and other owners' branches were untouched.

[ScriptedAlchemy]

Refs #4756 — new exact-head Bazel verification blocker; PR remains under merge hold.

Inspected master at adf02208ff40f4ae4fd8f975cb31ace772dd577a, the root AGENTS.md and full recursive tree (no nested AGENTS.md), the issue and both discussion comments, the existing issue-specific PR #4793 at exact head 9573a0333144fb90c928fbf4ba3b3703c4de9810, its changes/reviews, and the current CI attempt.

Observed failure (not an inferred assertion failure): Linux fleet job 129445801501 ran Bazel's //crates/tracedecay-global-db:unit_test, which reported TIMEOUT after 300.2 seconds. Its archived Bazel test log (bazel-test-logs-Linux-fleet artifact #13714603374) says 521 tests started; 505 printed ok, with no explicit failing Rust assertion before -- Test timed out --. One unrelated registered::workflow_schema_tests::published_artifact_ceiling::background_upgrade_is_atomic_at_every_cancellation_point test emitted a running for over 60 seconds warning; that does not prove which test exhausted the budget. Test Linux aggregate subsequently failed because its Linux shard failed. Clippy, repository gates, feature gates, dashboard and benchmark self-tests passed; Windows remained running at inspection.

Isolation needed: establish which of the 16 non-completing global-DB tests is holding the suite, especially whether the new real projected-record multi-page restart fixture (1,005 observations) is expensive versus an unrelated slow fixture or fleet contention. The existing 10,003 presence-only fixture is now seeded with 79 statements, but this run still timed out; the previous fixture optimization alone did not establish a passing global-DB suite. Avoid reducing the test's multi-page restart/rollback acceptance or increasing the timeout without measured attribution. Use focused, nonvacuous Bazel test execution and source diagnosis; Cargo is forbidden.

No new source change, CI weakening, merge, protected draft/Kev/#4776 change, release #4763 change, or other owner's branch/schedule mutation. Do not merge #4793 or close #4756. The Codex mid-directory durable continuation and incident-scale provider-attributed CPU / 125–160 ms profile-write improvements remain unproven on master.

[ScriptedAlchemy]

Refs #4756 — reviewed and published cd990101 as an exact-head, non-force fast-forward from 9573a033 on this existing PR.

CI diagnosis

I downloaded and hash-verified Linux artifact 13714603374 from run 42696633132: SHA-256 cd3793c7944ee3f83ce6ffeb557cee80d7d6c421eeb676d4be723d5c7c66e147. The global-db log starts 521 cases and prints 505 successful completions, with no explicit failed assertion before the 300-second suite timeout. Every retry-recovery case printed success. The workflow cancellation case with a >60-second warning subsequently printed success too. The final completions are source-revival cases; the archive does not establish whether the 1,005-record source-rewrite restart test had started, or identify it as the cause. No Windows log or new baseline attribution is claimed.

Published correction

observation_projection/rearm_tests.rs::seed_queue performed four serialized asynchronous SQL submissions per fixture row. It now submits four statements per explicitly ordered page of at most 128 rows. The canonical typed record constructor, exact serialized receipt/observation/cursor data, due/deferred distribution, auto-increment sequence/queue order, and one atomic setup transaction are preserved. Each statement additionally asserts its exact inserted-row count. The source/test diff contains only this fixture function: every existing recovery case, test size, production operation, writer-work assertion, CI deadline, concurrency setting and test count is unchanged.

For the 256-row fixture, setup falls from 1,024 SQL submissions to 8; the 260-row fixture falls from 1,040 to 12. These are statement counts, not a measured Rust wall-clock speedup and not proof that the complete suite now finishes within its limit.

Verification

An independent SQLite 3.46.1 differential reproduction compared the old scalar and new rowset SQL in 11 cases (empty, mixed due/deferred, exact page boundaries and multi-page fixtures through 514 rows). Every table's stored rows, sequence identities and ordering matched exactly, including escaped/unicode text. Foreign-key checks and integrity checks passed. A duplicate receipt injected on the final page of a 260-row setup rolled the entire transaction back, leaving all four tables empty. These are SQL mechanism checks, not repository Rust execution.

The detached Git commit was separately diff-reviewed before the expected-old-head force=false update; GitHub reports exactly one modified file and one function hunk. New exact-head Bazel CI 42719854458 is queued. This environment has no Bazel or checkout, so compilation, rustfmt/Clippy and the new-head full Rust suite remain unverified. No Cargo was invoked.

This removes avoidable cumulative fixture overhead, not a confirmed production re-import fix or proven resolution of the suite timeout. Keep the PR unmerged and #4756 open pending actual hosted results, durable Codex mid-directory continuation and incident-scale CPU/write-latency acceptance. Protected #4776/Kev drafts, release #4763, other owners' branches and task schedules were not modified.

[ScriptedAlchemy]

Refs #4756 — published a bounded source-rewrite tail fix to this existing issue-specific PR, without touching other work.

• 69742813: source-rewrite completion formerly selected at most 128 obsolete identities; if the remaining count was exactly 128 (or another multiple of 128), it could not know the page was final. It committed those retirements, then acquired an additional empty writer transaction solely to clear the pending marker. The query now reads at most 129 identities but settles/deletes at most 128, using the one-row lookahead to finish in the last productive commit. The existing full-source recheck after a resumed keyset page remains intact for concurrent older offers.
• 010fcce8: independent source review replaced an ignored Vec::pop() must-use return with truncate, retaining the same bound and avoiding a strict-warning gate.

Nonvacuous registered-store regression: seeds 128, 256 and 257 obsolete presence identities plus one current-generation survivor, runs the real completion API, and asserts exactly 1, 2 and 3 committed writer transactions, respectively, with all obsolete identities deleted, the survivor preserved, and the pending marker cleared. The former implementation fails the first two transaction-count assertions.

Independent SQLite reproduction (not Rust/Bazel): 128 obsolete rows took 2 → 1 transactions and 256 took 3 → 2; 127 and 257 stayed at 1 and 3, respectively. All cases preserved five current-generation survivors and retired every obsolete identity.

Both commits were published by exact-head, non-force fast-forward to #4793 (head now 010fcce8). Exact-head hosted Bazel CI #42720532582 is queued. Bazel is unavailable locally, so no new Rust test pass or timeout resolution is claimed. The preceding CI-timeout fixture optimization cd990101 remains intact and likewise requires fresh hosted execution.

Partial only. Durable Codex mid-directory restart continuation and incident-scale four-hour CPU / 125–160 ms writer acceptance remain unproven. Keep the PR unmerged and issue #4756 open. No protected drafts (including Kev/#4776), release #4763, CI policy, other owner's branches, credentials, tags, production/infra or schedules were changed.

[ScriptedAlchemy]

Refs #4756 — published c47732cf to existing issue-specific PR #4793, by exact expected old head 010fcce8, non-force fast-forward.

Additional write amplification root cause: retire_source_record requested a session-temporal reset for every newly retired observation. For a host source with many observations in the same session, this repeatedly ran INSERT ... ON CONFLICT DO UPDATE on the same session_temporal_resets row inside each 128-record source-rewrite writer page, even though a single per-session invalidation suffices. This was separate from the earlier page-boundary empty-writer and source-presence no-op updates.

Focused repair: per-observation provenance, ownership, disposition and projection retirement remain unchanged. The source-rewrite completion caller collects only successfully retired identities per page and calls the existing session-temporal store authority's new request_session_temporal_resets batch. It selects DISTINCT effect.session_id from those effect identities and performs one reset upsert per affected session; different sessions remain separate. The reset runs inside the same retirement transaction and 128-identity page, so a failed projection/reset rolls back that page, earlier committed pages remain durable, and the final rewrite marker is committed together. The singular reset API remains for other callers.

Nonvacuous registered-store regression: four real projected observations of one session are persisted, source presence is recorded, and the actual rewrite completion runs with a SQLite trigger that rejects a second UPDATE of that session's reset row. The old per-observation path would fail on observation 2. The new path must retire all four observations, create exactly one reset row, clear every stale presence identity, and clear the pending rewrite marker. Existing larger multi-page rollback/restart and per-record provenance regressions remain unchanged.

Independent SQLite verification, not Bazel evidence: 1, 4 and 128 candidate IDs with same/multiple session owners passed the new json_each + DISTINCT + UPSERT statement and produced exactly one row per session. The old repeated upsert fails under the regression's UPDATE-rejecting trigger. SQL used an in-memory disposable database; no real user data was touched. No end-to-end throughput, CPU or writer-latency result is claimed.

CI / merge hold: exact-head hosted Bazel run #42731245831 is pending. There is no Bazel workspace or Rust toolchain in this runtime, so this change has not yet passed Rust compilation, formatting, Clippy or the nonvacuous regression. Do not merge without these. Durable Codex mid-directory restart continuation and four-hour incident-scale, provider-attributed CPU/write-latency acceptance are still outstanding. Keep issue #4756 open, Refs only.

Reviewed the fresh root AGENTS.md; repository recursive tree had no nested AGENTS.md. Reviewed issue body and both comments, master and merged #4848, #4793's reviews and CI; all five prior review threads remain resolved. Protected #4776/Kev drafts, release #4763, other owners' branches and schedules untouched. No Cargo, Kody, Copilot, delegation, CI weakening, production, infra, tags or credentials.

[ScriptedAlchemy]

Refs #4756 — published 721fdc8a to this issue-specific PR by exact-old-head c47732cf, force=false fast-forward.

Focused regression-fixture correction: The 1,005-real-observation restart regression formerly performed its entire precondition through one seed_projected_messages(&runtime, 0..1005) call. That helper submits all 1,005 admissions in one call and asks project_queued_observations(1005) to project them inside a single writer transaction before source rewrite begins. This is an unnecessarily oversized fixture transaction and possible confounder for the earlier Linux 300-second full-suite timeout. The archive had no Rust assertion failure and did not identify this test as the actual timeout cause, so no such attribution is claimed.

The regression now calls the same production-path helper in eight ordered contiguous ranges of at most 128 observations, preserving the full 1,005 rows, canonical source cursor chain, final-page provenance collision, committed-page rollback, database reopen, and all existing durable-restart assertions. An explicit nonvacuity assertion checks all 1,005 records are seeded. No production source, CI deadline, threads, expected output, or test count changed.

Independent diff review verified exactly one test file and a single 9-line replacement; the seed_projected_messages helper calculates its expected cursor from indexes.start * 100, matching the previous window's final observation byte offset, and checks every admission/projection outcome itself. This is source-review evidence, not a Rust test pass. Local checkout/Bazel/Rust toolchain are unavailable; no Cargo was used. Exact-head hosted Bazel CI #42736729359 is queued.

Keep this PR unmerged and #4756 open. Durable mid-directory Codex restart continuation and the reporter's four-hour 1.5–1.8-core/125–160 ms write-latency acceptance remain unproven. Protected draft #4776/Kev, release #4763, concurrent owner branches and schedules were untouched.

[ScriptedAlchemy]

Refs #4756 — published 0099fe6b on this existing issue-specific PR, with an exact expected-old-head 721fdc8a, force=false fast-forward.

Confirmed CI blocker: Bazel run 42736729359 failed its Repository gates because the pinned //crates/tracedecay-global-db:rustfmt_check_2024 rejected one four-line formatting hunk in src/tests/source_rewrite.rs at the newly introduced 1,005-record restart fixture's assert_eq!. All eight repository-policy tests passed. This run never established the new-head global-db Rust test result; its later jobs were superseded.

Correction and independent review: the published commit applies the exact formatter output and changes one assertion's formatting in one file (+5/-1 lines), preserving its failure semantics, fixture count, bounded 128-record setup, restart/rollback tests and production behavior. Compared the full immutable parent blob, reviewed the detached commit's only file patch, and verified GitHub reports the intended new PR head. No tests or timeouts were loosened; no Cargo was used.

New exact-head Bazel run 42742041492 is queued; no Rust test/Clippy pass on this head claimed. Durable Codex mid-directory restart continuation and incident-scale CPU/write latency remain unproven. Keep PR #4793 unmerged and #4756 open until those requirements pass on master. Protected #4776/Kev drafts, release #4763, other owners' branches and existing schedules are untouched.

[ScriptedAlchemy]

Refs #4756 — new exact-head Bazel CI blockers, not yet a proven source regression (2026-11-10).

Reviewed current master da7a6fdde6f0cd86c7affc9a8d58297e20ae8d95, the root AGENTS.md and full recursive tree (no nested AGENTS.md), issue body/all two comments, existing issue PR and its five resolved inline review threads, all 25 current changed paths, the source-rewrite/temporal-reset implementation, and related Codex/other-owner PRs. No conflicting branch was edited.

At exact PR head 0099fe6b6ebd3e60118b68be6da9348814b73be7, hosted Bazel CI #42742041492 has passed Repository gates (including the previously failing pinned Rust formatter), Clippy, Dashboard, Feature gates, and benchmark checks. However the Linux fleet job failed two distinct targets; its uploaded test artifact 13735182684 is associated with this exact head, with digest sha256:3c2b7fcde27e22e7a8453da3e0df843f56dc08c334e678c4a816df502918328a.

1. //crates/tracedecay-global-db:unit_test timed out at the unchanged 300-second limit. The raw Bazel log starts 523 cases and reports 338 explicit ... ok completions with no Rust assertion failure before the timeout. registered::workflow_schema_tests::published_artifact_ceiling::background_upgrade_is_atomic_at_every_cancellation_point warned after 60 seconds but eventually passed, so that warning is not evidence of a permanently stuck test. The prior exact-head timeout had 505/521 completions; this run establishes significant total-suite variance, not which remaining test caused it or whether the 1,005-record source-rewrite regression was even executed. Do not increase its timeout, weaken tests or attribute failure without isolated nonzero runs.
2. //crates/tracedecay:mcp_suite ran 675 cases: 674 passed, one failed, specifically mcp_handler_test::graph_query_test::test_module_api. At tests/mcp_suite/support.rs:570, a 20,001-ms production-composition readiness wait elapsed while code-index freshness was Indexing/rebuild in flight and its phase was Verification with 3/3 files, 15 chunks, 5,633/5,633 lexical units complete; mounted_generation=None. This is a real admission/readiness failure, not a compiler or formatting failure. The MCP fixture/graph-publisher files are not changed by perf(sessions): bound projection queue-head scans per write window #4793. Baseline/host contention versus a cross-subsystem regression is not proven; coordinate with its owner rather than altering unrelated acceptance paths.

The published issue-specific PR still includes bounded source-presence/rewrite/retry work and restart regressions, but no exact-head full Bazel pass or reporter-scale 1.5–1.8-core / 125–160-ms write-latency proof. Durable Codex mid-directory position also still disappears on daemon restart; the whole-sweep V2 frontier cannot claim intermediate continuation. That requires a separately justified safe durable state/identity design, not an invented scan-resume claim.

Merge/closure HOLD: #4793 remains unmerged and #4756 stays open; Refs only. Windows and Linux aggregate had not completed at last check. The blocker is now isolating the timed-out global-db test(s), distinguishing the MCP readiness failure from baseline load, and obtaining nonvacuous Bazel behavior on the exact source head. No Cargo, CI weakening, speculative push, forced branch movement, drafts/Kev/#4776 merges, release #4763 changes, production/infra/tags/credentials, other-owner edits or schedule changes.