Repository navigation
fix(code-graph): retain authenticated proofs across interrupted successors - #4846
ScriptedAlchemy wants to merge 12 commits into
Conversation
Reapply the six exact issue-owned source and regression blobs from #4791 at 22c534b, without that branch's draft CLI-fixture ancestors. Keep authenticated snapshots across interrupted and overlapping proofs, real one-file successor and corruption regressions, and the validated shutdown and graph-tool corrections from the prior review lane. The original history and draft prerequisite remain intact on their own branches; this single-parent commit starts at master 66963c8. Exact-head Bazel and human review remain required. Refs #4507
|
#4507 exact-head CI blocker — 2026-11-09 (no merge)Read current root AGENTS.md and recursive master tree (no nested AGENTS.md), all 19 issue comments, current master proof implementation, this six-file patch, related #4523/#4520/#4789, reviews and the actual Linux fleet logs. The branch remains at New exact-head failure: Bazel CI #18004 / Linux fleet job 129089747041 completed 182/183 Bazel targets passed; 68 actually executed. Repository gates, scope checks, benchmark harness, Clippy, Dashboard, and feature gates passed. Windows native build/type-check is still running; its native test step has not executed. macOS native is skipped. Do not cancel the Windows run just to retry; once the workflow completes, a targeted Bazel failed-job rerun or an independently executed nonzero isolated reproduction is appropriate, with the real failure investigated if it recurs. Merge hold remains pending a fully green exact-head run, independent human approving review, native sealed-proof tests where available, and reproducible real-checkout before/after one-file-edit wall time + peak RSS. No real-checkout benchmark exists in this environment, so none is claimed. Keep |
An unchanged overlapping proof replaced the shared Arc with an equivalent map, so a concurrently authenticated one-file successor lost its snapshot CAS and discarded its new fingerprints. Retain snapshot identity when no fingerprint differs and the proven key count is unchanged, without adding a full map comparison. Changed and deletion-only proofs still use the existing exact-snapshot CAS. Add real-file regressions for the unchanged-proof/successor interleaving and deletion-only pruning, including an empty result. Strengthen stale commit coverage with a genuinely new authenticated segment. Existing corruption, source-identity, extent and cancellation checks are unchanged. Refs #4507
|
Published review repair Concrete defect: an unchanged parent proof always allocated/published a replacement The one-file production repair preserves the snapshot when all recorded fingerprints equal the parent and the distinct-key count is unchanged. Genuine additions/replacements and deletion-only updates retain the existing exact-snapshot CAS. No second full map scan, source check removal, or non-Unix behavior change. Added two real-file regressions: unchanged overlap must preserve the authenticated edited segment and prune the replaced one; deletion-only/empty proofs must still remove old entries. Strengthened the existing stale-contender test with an actually new authenticated segment so it must exercise the changed-state CAS rather than the no-op branch. All previous corruption/fingerprint/cancellation/extent assertions remain. Hosted Bazel run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42719106126 (#18222) is pending. The new Rust tests have not yet executed here; local Bazel is unavailable. The preceding head's Linux retry is green but does not validate this commit. MERGE HOLD remains for exact-head nonzero validation, independent review, native tests when available, and representative wall-time/peak-RSS evidence. Sweep coordination: do not duplicate this fix or import #4796's separate |
The daemon retained a full authenticated generation map for every root ever visited, even after that root stopped being used. Keep only the most recently used 32 root snapshots; an evicted root safely rehashes when revisited. Recent successful reads refresh recency without changing the snapshot Arc or the overlapping-proof compare-and-swap. Commits remain conditional on the exact parent snapshot, so eviction cannot promote in-flight stale proof. Add a real-file multi-root regression proving the cap, LRU eviction, retained recent-source reuse and new-root source proof. No bypass of file identity, manifest extent, digest or corruption checks. Refs #4507
|
Published a source/test repair on existing #4846: Confirmed cache-retention defect: Fix: retain the 32 most-recently accessed roots, keeping the same shared immutable per-root snapshot and pointer-CAS for concurrent proofs. Reading a root refreshes recency without replacing its New nonvacuous real-file regression constructs 32 separate authenticated roots and an additional incoming root. It asserts the exact retained-root cap, fresh-hash fallback for the oldest unused root, and positive authenticated reuse for a recently read root and the incoming root. Previous same-file corruption, replacement, extent, cancellation and stale-CAS tests remain unchanged. No dependencies, workflow edits or sibling #4520/#4532 source paths. Validation: new exact-head hosted Bazel run #18286 queued; no new-head Rust test/formatter/Clippy success yet. Predecessor #18222 actually executed the store-runtime suite successfully, but cannot prove this new code. Local Bazel and the representative real-checkout workload are unavailable; no Cargo/standalone Rust commands used. Representative before/after one-file-edit wall-time/peak-RSS, native proof execution and independent approval remain unproven. Preserve merge hold and keep #4507 open. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The bounded root cache evicted an authenticated parent while its one-file successor was still proving, so the successful successor lost the snapshot CAS and the next edit had to rehash the whole generation. Skip entries whose authenticated snapshot is held by an in-flight proof; evict only idle roots and preserve the 32-root bound. Keep the stale-proof guard, corruption checks and cold-root reauthentication unchanged. A real-file 33-root regression makes an edited root least-recently-used while its successor is in flight, and verifies inherited and changed source stay warm, replaced source stays cold, and an idle root is evicted. Also apply the exact Bazel rustfmt layout required by current-head gates. Refs #4507
When all 32 cached roots have in-flight proofs, admitting a newly verified root evicted the only idle candidate: the new proof itself. That made the next one-file edit rehash the entire generation. Retain the newest authenticated root temporarily while older parents are pinned; reclaim excess idle roots as the final proof references drop. Keep stale-generation fencing and source identity checks intact. A 33-root real-file regression demonstrates warm source survival, transient over-cap retention and eventual return to the 32-root bound. Refs #4507
|
Published source + real-file regression repair Newly demonstrated performance correctness defect: if 32 distinct roots all have active proof references, their snapshots cannot be evicted without invalidating their successor CAS. The previously implemented strict 32-root eviction then selected the newly successful 33rd root as the only idle victim. Its freshly authenticated generation was immediately dropped, so the next one-file edit rehashed all source again. Repair: LRU trimming now preserves the most recently authenticated root while all older entries are pinned; temporary over-cap entries are bounded by outstanding active proof pins. Falsifiable regression: Exact-head hosted Bazel CI #18334 is pending. No Rust/test pass is claimed for the new head. Local checkout/Bazel and the original ~6.1k-file profiling corpus are unavailable here, so representative before/after full-pass wall-time, source-read and peak-RSS acceptance remains unproven. Existing DO NOT MERGE or close #4507 yet: exact-head nonzero Bazel/native evidence when available, independent human approval and representative same-corpus measurements remain required. This edit did not touch other owners' #4520/#4789 or #4532/#4796 paths, or protected decision-model/release work. No Cargo, force, duplicate PR, test weakening, or infrastructure change. |
The daemon can begin indexing on a filesystem edit before the explicit sync request. Starting wall time and peak-RSS observation after the source write misses work triggered by the edit. Arm both before writing. Exercise the real harness edit path with a deliberately slow write: require RSS observation before mutation, full wall-time inclusion, and correct readiness and peak-memory attribution. Preserve existing full-run fixture integration coverage. Refs #4507
|
Published issue-owned benchmark measurement fix The patch moves timer and sampler activation ahead of the source mutation and keeps stop after the observed new generation is ready; sampling is stopped in Published safely via exact expected old head |
An asynchronous daemon refresh can advance the current generation after initial indexing but before the benchmark edits source. Using the old initial-generation ID could incorrectly accept that prior refresh as the result of the edit, reporting a tiny wall time and missing peak memory. Read and require the latest current, ready generation before mutating the file; only a subsequent nonempty generation ID ends the timed pass. Preserve the sampler-before-write timing and update the observed generation. Exercise a real tracked edit against a deliberately stale index-lane ID, reject a missing pre-edit ID without modifying source, and refuse a false ready result with an absent generation until a real successor appears. Refs #4507
|
Published e4064bbcb350 on existing #4846 ( Actual measurement-correctness defect: the one-file benchmark waits for a generation different from the ID saved at initial indexing, but the daemon can complete another background refresh before the measured edit. The pre-existing generation ID then becomes stale; the old implementation could accept that unrelated already-completed refresh as the edit's result, reporting an artificially short pass and missing incremental peak RSS. A response claiming current/ready with an absent generation ID could also satisfy the old Repair: require a nonempty, current/ready generation ID immediately before mutating the tracked source, use that freshly observed ID as the baseline for the edit-to-ready wait, require a nonempty successor ID, and update the tracked generation after the measured pass. Existing timer/RSS observer still starts before the file mutation and stops after readiness, preserving the previous fix. Falsifiable tests: the existing real tracked-file slow-edit regression now supplies an obsolete initial generation and an independently newer pre-edit generation; it asserts the wait uses the newer ID, includes the deliberate edit latency, and records peak RSS. It also requires a missing pre-edit generation to refuse without changing the source or starting sync, and a missing post-edit generation to be skipped until a later valid successor arrives (two status polls). Previous full fake-daemon harness tests remain unchanged. Validation: new exact-head hosted Bazel CI #18380 pending; no current-head test result claimed. No local checkout/Bazel was available, so this is a reviewed patch awaiting execution, not a benchmark run. Representative same-corpus baseline/head one-file-edit wall time, actual source-read/hash work and peak RSS, independent human approval and applicable native proof tests remain unproven. Preserve merge hold; do not duplicate this issue PR, change protected decision-model/release lanes or weaken validation. |
A tracked source symlink can target a file outside the disposable checkout; appending to the first git ls-files result could mutate operator source. Git's quoted filenames also break whitespace splitting and abort real runs. Select only regular non-symlink files resolved within the private clone, and read NUL-delimited Git paths to retain spaces and newlines. Exercise the full fake-daemon benchmark with a committed external symlink and a spaced Rust source path. Require that only the safe path is selected and the external original remains unchanged. Preserve existing timing and RSS sampling behavior. Refs #4507
|
Published actual benchmark safety/reliability repair Concrete defect: the one-file edit benchmark appended to the first Fix: use NUL-delimited Git output, decode each pathname unchanged, and edit only a regular, non-symlink file whose resolved path remains inside the disposable clone. Preserve the existing pre-edit timer/RSS sampling, baseline generation fence, new-generation wait, and strict error handling. Falsifiable regression: a real temporary Git repository commits an external-target symlink as the first candidate plus a spaced-name Rust file. The full fake-daemon benchmark must select the safe regular file, leave the external original untouched, preserve the original checkout, and clean its temporary run. On platforms without symlink fixture support, the spaced-path control still runs. Prior source would fail this test; it either errors on the quoted path or follows an unsafe candidate. Validation boundary: hosted exact-head CI #18405 is pending. No passing new-head Python/Bazel/Rust tests or representative wall-time/peak-RSS measurements are claimed; no local checkout/Bazel was available. Independent approval and native tests when available remain required before merge. Issue/PR remain OPEN and unmerged. Separate #4520 and #4532 owners are unaffected; no Cargo, workflow change, force push, protected-model/release or production/infra action. |
Use the complete typed status result before the CLI's possibly truncated text preview. Make the existing end-to-end fake-daemon regression emit preview-only text with real readiness in structuredContent, so the former parser fails before any generation becomes ready. Refs #4507
|
Published issue-owned benchmark readiness fix Strengthened the existing real-Git/fake-daemon end-to-end benchmark regression: its successful status now deliberately returns complete freshness, generation, and memory only under Verified GitHub branch/PR both at |
Refs #4507
Exact-head CI blocker — November 10, 2026 (00:06 UTC)
At current head
43e845887c57e8c36302cd47c1446c82cb06d537, hosted Bazel run #18222 failed Linux fleet 1/183 targets, with 182/183 passing and 64 targets actually executed. The affected//crates/tracedecay:runtime_acceptance_suiteexecuted 140 passed / 1 failed;runtime_surface_acceptance::production_lsp_negotiates_and_projects_canonical_contextreportedDeadlineExceededwhile acknowledging a daemon LSP frame atruntime_surface_acceptance.rs:433. Exact Linux job. The LSP test is outside the six changed paths, and related failures have occurred historically, but whether this instance is a baseline timing failure is not proven. Do not weaken its deadline or assume that a retry establishes root cause.Issue-specific result:
//crates/tracedecay-store-runtime:unit_testPASSED in 54.8s (not cached), exercising the head containing both newly added real-file regressions and the strengthened stale-CAS test. This is non-vacuous target execution, not end-to-end one-file-edit performance proof. Repository gates, Clippy, change scope, Dashboard, feature gates and benchmark-harness checks passed. Windows build/type-check has since completed successfully; its native-test step and ordinary-PR macOS tests were skipped. With every job completed, a failed-job-only rerun was requested against this unchanged head (GitHub run #18222, attempt 2; Linux fleet queued at 2026-11-10 00:54 UTC). This is a retry, not a pass; keep the branch stable while it runs.Required before merge: attribute or fix the LSP acceptance failure without lowering coverage, obtain independent human approval, run native sealed-proof tests when available, and measure real-workload before/after full-pass wall time and peak RSS. No representative checkout with Bazel or native workflow dispatch was available in this run. Keep
Refs #4507and the issue OPEN.MERGE HOLD: Do not merge/auto-merge or close #4507 from partial evidence. Require nonzero exact-head Bazel results, independent approving review, relevant native tests when available, and representative before/after one-file-edit full-pass wall time and peak RSS. This remains partial work; no test assertions, deadlines or protections are relaxed.
Current head and new review repair
Head:
43e845887c57e8c36302cd47c1446c82cb06d537, targetingmasteron the existingfix/4507-sealed-proof-masterbranch. The latest commit is a direct child of1274238745ce0f529be83161ab528a5461cca67f, published with that exact expected head andforce=false. It changes onlycrates/tracedecay-store-runtime/src/session_registry/verified_sealed_segments.rs; the other five existing PR files are preserved.Unchanged overlapping proofs discarded a valid successor
Review found a remaining race in the authenticated-snapshot cache. Given a parent snapshot with two authenticated segments, a one-file successor starts from it and authenticates its changed segment. An overlapping recheck of the unchanged parent then commits first. Previously, that no-op recheck replaced the shared
Arcwith an equivalent map, causing the successor's pointer compare-and-swap to fail. Its newly authenticated source was discarded and obsolete parent entries remained cached.The cache now records whether any observed fingerprint differs from its starting snapshot. No new fingerprints plus the same number of distinct proven entries establishes an identical map without a second full-map scan. Such a completed proof preserves the existing snapshot identity instead of publishing a replacement. Real changes and deletion-only proofs still use the existing exact-snapshot CAS; a genuinely stale changed proof cannot overwrite a newer successful proof.
This changes cache publication only. Every reuse still checks the physical file fingerprint and declared manifest extent. All source/manifest verification still has to succeed before
commit; aborted/corrupt proofs cannot publish tentative hashes. Unsupported non-Unix platforms still rehash source.Falsifiable regression coverage in the new commit
unchanged_overlapping_proof_preserves_authenticated_successor: real nonempty SHA-256-authenticated files, deterministic overlapping-proof order, a positive changed-source reuse assertion after commit, and refusal to reuse the replaced segment. The old unconditional snapshot replacement loses the successor and fails the positive assertion. Parent and successor have equal segment counts, so a count-only shortcut also fails.pruning_without_new_fingerprints_still_replaces_the_snapshot: a deletion-only successor must keep the remaining authenticated segment but drop the removed entry; a subsequent empty proof must drop the last entry. This guards against treating all no-new-hash proofs as no-ops.overlapping_proofs_share_parent_and_stale_commit_cannot_replace_successorwith a newly authenticated segment in the stale contender. The test still requires the winner to remain reusable and rejects both resurrected parent entries and that abandoned new segment, exercising the genuine-change CAS path.The existing cancellation/tentative-isolation, inode replacement with restored mtime, root/extent/deletion, real partitioned one-file successor, changed-source corruption, repaired-source rehash and warm recovery assertions remain intact. No fixture, production fingerprint, manifest format, dependency, workflow or infrastructure change.
Original sealed-source fix
Merged #4523 removed routine whole-generation source rehashing. This PR fixes the remaining cancellation/overlap cold-cache paths:
VerifiedSealedSegmentsV1::beginpreviously removed the sole authenticated map. Retain the last fully authenticated snapshot in an immutable sharedArc, keep new observations private, and reject stale changed completion at commit. Reuse requires matching device/inode/size/mtime/ctime and manifest extent; non-Unix retains full-hash fallback.The six-file PR also preserves the existing, separately committed CI repairs from #4791: admitted graph-read timeout/cancellation classification, early session-refresh cancellation, and arming every project's feedback shutdown before awaiting individual workers. Their regression coverage and original daemon lease assertions are unchanged by the current one-file fix. Sparse-successor work remains separately owned by #4520/#4789; graph-open work remains in #4532/#4796.
Validation and evidence boundaries
Current-head Bazel test evidence: run #18222:
//crates/tracedecay-store-runtime:unit_testpassed in 54.8s, not cached, under a 182/183 Linux-target run; onlyruntime_acceptance_suitefailed due to the LSP DeadlineExceeded described above. Repository gates, Clippy, Dashboard and feature gates passed. This is code/regression validation, not end-to-end performance evidence.Predecessor evidence only: run #18157 tested
1274238745ce. Its Linux retry job #129478492894 now reports success. The first attempt had 182/183 Bazel targets passing, with the sealed-proof target passing and a dashboard HTTP timeout; a successful retry does not establish that timeout's cause. Prior Windows build/type-check passed but native tests were skipped; ordinary PR macOS was skipped. These results do not validate the new commit.No Bazel/Bazelisk or repository checkout is available in this execution environment. The edited source was checked against the exact original Git blob and the published one-file commit diff; those are integrity/source-review checks, not Rust execution. No Cargo, direct rustc or direct rustfmt was run.
Native macOS/Windows sealed-proof tests require the existing workflow's explicit dispatch (
run_os=true,native_platform=all,native_targets=//crates/tracedecay-store-runtime:unit_test). No native dispatch or native pass is claimed. Independent review remains outstanding; the earlier bot COMMENTED review of52bac326is not an approval of this new head. Historical #4523 source-proof measurements and suite durations are not representative before/after full-pass time/RSS measurements for this repair.Review boundary and coordination
This remains the sole active issue-specific replacement for closed, unmerged #4791. At creation, six exact issue-owned file blobs were applied directly to master
66963c8bb1d44816b5c1f7aeb67a53458ea83dfd, avoiding the unreviewed CLI-fixture changes and draft #4834/#4837 ancestry accidentally introduced to the old lane. Subsequent ordinary master merges preserved that six-file boundary. Old branches/history remain preserved. The current source fix does not import another lane or resync unrelated master changes.Sweeps: use this existing PR, preserve its head while the new CI runs, and do not duplicate or merge it from predecessor evidence. Keep
Refs #4507; leave the issue open until every acceptance criterion is demonstrated on master. All Kev/decision-model work, including mixed #4776, remains draft-only and must never merge/auto-merge; release #4763 is read-only. No force/rewrite/discard, external delegation, releases/tags, production, credentials or infrastructure work is included.