Repository navigation
fix(code-graph): retain sealed proofs across interrupted successors - #4791
ScriptedAlchemy wants to merge 16 commits into
Conversation
|
|
Coordination / merge hold for #4507: PR #4791 is an issue-specific regression for the production fix already merged in #4523. I am marking this test PR ready for review solely so the normal hosted Bazel Linux CI runs (the repository skips that authority for drafts). Do not merge or auto-merge #4791 in the general PR sweep yet. Required before merge: exact-head nonzero Rust regression results, formatting/clippy where applicable, independent review and a verified absence of overlapping fixes. Keep issue #4507 open until the end-to-end memory/time and corruption acceptance is proven on |
|
#4507 exact-head CI repair (2026-11-08 UTC): Hosted Bazel run #17255 found one actionable failure: |
Keep the last successful sealed-segment proof in an immutable shared snapshot. Starting, cancelling, or overlapping another proof must not evict it or expose uncommitted hashes. Publish a completed proof only if its starting snapshot is still current so a late proof cannot replace a newer successful proof. Add real-file regressions for cancellation, tentative hash isolation, overlap, stale commit rejection, same-size replacement with restored mtime, wrong root/extent and deleted source. Existing one-file successor and corruption regressions remain unchanged. Refs #4507
ScriptedAlchemy
changed the title
|
Published source/test fix Root cause: Four real-file regressions cover positive parent reuse after abort, tentative-hash isolation, concurrent reuse/stale completion, same-size replacement with restored mtime, and root/extent/missing-source refusal. Existing one-file-successor/read-count and corruption regressions are preserved. This is source engineering, not a status-only update. Exact-head hosted run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442313648. Repository gates, scope detection, and harness checks are queued; native macOS is skipped. No successful new test result is claimed. Bazel/Bazelisk is absent locally; no Cargo or substitute Rust execution was used. MERGE HOLD remains. Independent review and fresh full-pass time/peak-RSS evidence are still missing. This author pass is not independent review. Sweeps: keep this issue-specific PR, do not duplicate/merge it from a partial green, and leave #4507 open. No #4520/#4789 implementation, Kev/decision-model work, release #4763, workflow, credentials, or infrastructure changes are included. |
After a real changed-segment corruption refusal and restoration, require a nonzero re-verification of the repaired source while proving that untouched inherited segments remain authenticated. A cold recovery would fail the existing generation read-count contract rather than passing merely because integrity checks still reject corruption. Refs #4507
A positive proof callback count is vacuous because verifying the manifest and completing the bundle also invoke the callback without reading changed segment bytes. Measure a successful fully warmed successor first; after corrupting and restoring its changed file, require strictly more checks than that warm baseline, then confirm a subsequent pass returns to baseline. Keep the cold-versus-inherited proof bound and typed corruption assertions. This adds no production behavior or CI bypass. Refs #4507
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
Published regression hardening |
The changed-segment recovery count is strictly greater than the warm baseline only where file identities permit inherited digest reuse. On unsupported platforms every pass reads all segments, so its counts are equal; corruption refusal and Unix regression remain strict. Refs #4507
|
#4507 exact-head CI update, commit Overall Linux CI remains red: 3 Keep explicit MERGE HOLD. Independent review and full-workload wall-time/peak-RSS remain missing, and failed Linux targets must not be bypassed. No new code change, force push, duplicate PR, or scope expansion. |
Reuse only the scheduler fixture repair from 2ecbdf1 in PR #4796. The old jobs.rs blob matches this branch byte for byte; no graph-reader changes or unrelated branch history are imported. Bind scheduler fixtures to the required backend identity while keeping schedule, lock, ledger and zero-call assertions. Retain the added unconfigured refusal, deduplication and same-occurrence recovery test. The identical repaired blob passed 171 automation-runner tests in hosted Bazel run 42489433681; this new head still requires its own validation. Refs #4507
The rename-preview deadline regression observed a real source read before expiry, but the inner dispatch timer returned a raw project-route error. The generic error mapper then mislabeled the admitted timeout as an unavailable authority before admission, depending on which timer won. Translate raw dispatch deadline and cancellation errors at the admitted read-owner boundary. Preserve authoritative problem records and typed details, side-effect settlement, and all other refusal classifications. Add deterministic regressions for the rendered terminal kind and stage, owner-record precedence, and effect/other-error isolation. Keep the existing paused-source-read end-to-end assertions and deadlines unchanged. Refs #4507
|
#4507 new exact-head CI blocker — 2026-11-08 UTC, PR #4791 at
MERGE HOLD remains pending clean/attributed CI, independent human review, native tests when available, and reproducible one-file-edit full-pass wall-time/peak-RSS measurements on a real checkout. |
Use the refresh registry's synchronous cancellation surface when preparing the engine's shutdown owners. Waiting to signal refresh inside its later join leaves lease-owning supervisors running while earlier invocation owners consume the common shutdown deadline. Add a real registered-project worker regression that observes cancellation before any prepared join is polled, discards those joins, and explicitly joins the registry before asserting the recorded result. Restoring the no-op cancel callback fails that positive cancellation assertion. Keep the existing daemon lease regression, owner order, deadlines and store-closure checks unchanged. Exact-head hosted Bazel is still required. Refs #4507
|
Published The #17542 artifact's failing daemon test is The new real registered-project-worker regression captures cancellation immediately after preparing shutdown, before any join is polled, discards the prepared joins, then drains the registry and requires zero workers. The captured positive cancellation assertion fails under the old no-op callback. No test-only production hook or mock owner was added. New-head hosted Bazel: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42525678986 (#17581), queued when inspected. No local Bazel/Bazelisk is available and no Cargo/direct Rust execution was used. The earlier 182/183 passing targets are predecessor evidence only. This patch fixes late cancellation, but the unchanged daemon integration test must pass before claiming the whole lease failure resolved. Merge hold remains. Sweeps should coordinate here rather than duplicating the fix. Independent review, native testing and fresh workload wall-time/RSS remain outstanding; #4507 is not closed. No Kev/decision-model, release, workflow, infrastructure or credential changes. |
|
#4507 follow-up on the existing PR (2026-11-08 UTC): The branch has advanced by a single non-rewritten child commit from Hosted Bazel run #17597 is queued for exact head |
|
#4507 exact-head CI diagnosis and branch integration (2026-11-09 UTC). The now-finished Linux Bazel fleet #17597 at
I then safely integrated current Current new-head hosted Bazel #17719 is queued, not tested yet. macOS native skipped by default; old Windows build cancelled and native Windows test step skipped. Independent approving review and fresh real-workload full-pass time/peak RSS still missing. MERGE HOLD: no merge, no #4507 closure, Refs only. General PR sweep: coordinate on this existing PR, avoid duplicate fixes and protected draft #4776; release #4763 read-only. |
|
#4507 exact-head hosted Bazel CI blocker — November 9, 2026 UTC. Reviewed the completed Linux-fleet job and downloaded its actual nonzero test-log artifact for PR #4791 head 182/183 Bazel targets passed; one target failed. Direct artifact evidence: New blocker: Repository gates, scope, Clippy, Dashboard, feature gates and benchmark harness passed. macOS test job skipped. Windows job remained running when inspected; its native test step must be verified separately. The overall workflow had not concluded at inspection. MERGE HOLD: Keep PR open and |
|
#4507 verification / stacking blocker (2026-11-09 UTC, exact head New target/scope blocker: #4791 currently targets the draft CLI fixture PR #4834 ( MERGE HOLD and |
Reapply the six exact issue-owned source and regression blobs from #4791 at 22c534b, without that branch's draft CLI-fixture ancestors. Keep authenticated snapshots across interrupted and overlapping proofs, real one-file successor and corruption regressions, and the validated shutdown and graph-tool corrections from the prior review lane. The original history and draft prerequisite remain intact on their own branches; this single-parent commit starts at master 66963c8. Exact-head Bazel and human review remain required. Refs #4507
|
#4507 replacement / coordination (2026-11-09 UTC): #4846 is now the only master-target candidate, from a clean separate branch at commit |
Refs #4507
Current integration and validation hold — November 9, 2026 UTC
Exact PR head:
22c534b70069dc0cbce776de38436d2359faff25, basecodex/cli-transport-fixture-admission(draft #4834), not master. Current master isbe2e1349c63f6834922a23c2f12ac8002431d99f, seven commits ahead of the shared merge base83694e0. Draft #4834 advanced toec0b338d8d41c836f97dd74bad69a22f42a4f23bafter this PR's last integration. The PR's six-file diff against the stacked base is issue-owned; a retarget to master today would also present the two CLI fixture files owned by #4834. Do not retarget in a way that imports unapproved draft work; do not merge or auto-merge this PR while stacked. Preserve both branches' history, coordinate the prerequisite independently, then restore a master-target review boundary without force/rewrite/discard.Exact-head hosted Bazel CI #17797: Linux fleet passed 183/183 targets (20 executed in this run, remainder cached); repository gates, Clippy, Dashboard and feature checks passed. The Windows job completed successfully only for build/type-check: its actual
Build and test native targetsstep was skipped. macOS native job was also skipped. Neither green Windows job status nor Linux cache hits establish native Windows/macOS sealed-proof execution. Earlier source-identical Linux runs executed the five proof/corruption regressions non-vacuously.Still missing: independent approving human review (only COMMENTED automated review recorded), fresh before/after real-checkout one-file-edit full-pass wall time and peak RSS, native-platform tests when supported, and correct master-target scope. No test assertions, deadlines, protections or hosted runners were altered.
Refs #4507only; issue open. Protected Kev/decision-model work (including mixed draft #4776), release #4763, and all production/credentials/infrastructure remain out of scope.MERGE HOLD: Do not merge/auto-merge or close #4507 from partial evidence. The exact stacked-head Linux fleet passed, including both previous CLI shutdown failures; independently approving human review, real one-file-edit full-pass wall-time/peak-RSS comparison, and native-platform tests when available remain required. Do not weaken assertions, raise deadlines or bypass protections.
Current CLI verification
This PR is stacked on #4834 for the common CLI startup fixture. The review diff against that base contains the existing sealed-proof and shutdown changes. The prior Linux job passed 204 of 206 CLI cases. Its two failures occurred while waiting for the child process to exit after SIGTERM. The socket-permission test had waited only for the inode. It now waits for the daemon to report readiness before asserting mode 0600 and sending SIGTERM. The in-flight-open case already waits for a held store boundary and shutdown draining.
Native Bazel passed both cases three times on the combined head
22c534b700, with the original three-second exit bound. The same two tests also passed three times on the prior native head. These results establish fixture behavior locally. Both previously failing CLI tests also passed in the exact-head hosted Linux fleet.Sealed-source fix
Merged #4523 removed routine whole-generation source rehashing. This PR fixes the remaining cancellation/overlap regression:
VerifiedSealedSegmentsV1::beginremoved the sole authenticated snapshot, so interrupted or overlapping verification made an unchanged generation cold again.Retain the last fully authenticated snapshot in an immutable shared
Arcmap. New observations remain private until complete verification succeeds. Reuse still checks file identity (device, inode, size, mtime, ctime) and the declared manifest extent; snapshot-identity compare-and-swap prevents stale completion from replacing a newer proof. Unsupported non-Unix fingerprints still take the full-read fallback. No manifest/digest format or dependency change.The two store-runtime files retain the real one-file successor regression, nonzero inherited-segment/read-count assertions, same-length corruption rejection, repaired-source rehash above the warmed Unix baseline, warm recovery, forged extent refusal, and four real-file abort/overlap/replacement/isolation regressions. Separate sparse-successor work remains in #4520/#4789.
CI blockers repaired in separate commits
5da6499a531c46accae4c2651ea156b926e13b5f— start feedback drains before per-project worker joinsIn daemon shutdown, each project's feedback observation worker was previously joined before the next project's terminal drain was signalled, serializing shutdown work and consuming the common owner grace period under many open projects. The source change separates synchronous admission closure and terminal enqueue from the retained asynchronous worker join.
shut_down_feedbackconstructs all project drain futures first (thereby sending every terminal record), then awaits those joins. Existing sticky error handling, terminal record persistence, lease-ownership joins and deadlines remain in place. The feedback cancellation regression was adjusted to assert admission closes before its join can run; the preceding engine shutdown cancellation regression remains.This is a plausible second cause of the 10-project daemon lease failure, not yet a proven repair of the end-to-end failing test. No benchmark or native test result is claimed for this commit. Its two-file production diff was reviewed against the parent and no issue-specific competing PR was found.
22129604ffdaea8f00dbf34c78e98c045e1fa986— cancel session refresh before shutdown joinsThe downloaded #17542 daemon-suite log fails
store_shutdown_checkpoint_test::capacity_retirement_behind_a_parked_bind_eviction_cleanup_completesat its store-close assertion. It reports unfinishedinvocationandsession_temporal_refreshowners and outstanding client leases. This remains a real failure, not a waived baseline or assumed flake.Source review found a concrete cancellation-wiring defect in
daemon/engine/shutdown.rs: the session-refresh owner had a no-op cancellation callback and only signalled its supervisors inside its later join. The coordinator broadcasts cancellation during preparation before polling any join; invocation cleanup can consume the shared grace period before refresh is even told to stop. The bootstrap already uses the registry's synchronouscancel_supervisorsauthority.The engine now supplies that same cancellation callback, retaining the existing join, store release, owner ordering and all deadlines. This fixes the late-cancellation mechanism; it does not yet prove every cause of the observed lease failure is eliminated.
New real registered-project worker regression
daemon::engine::shutdown::tests::session_refresh_is_cancelled_even_if_shutdown_joins_are_droppedrequires one live worker, captures its cancellation state immediately after production shutdown preparation and before any join is polled, discards the prepared joins, then explicitly joins the registry and checks its worker count is zero. The captured positive cancellation assertion fails with the former no-op callback. Cleanup occurs before that final assertion so the falsifying implementation does not intentionally leave a worker behind. No mock shutdown owner or source-shape assertion is used. The original daemon integration regression remains unchanged.36e6ab4499979863fb3aed1860362ccfb49d82d4— scheduler fixture repairReuse ONLY
crates/tracedecay/tests/automation_runner_test/jobs.rsfrom existing #4796 fix2ecbdf1d7645d9bca58fe589001b75c109cce7cc. Before-blob identity was checked against this branch; no graph-reader code/history was imported, and coordination was posted on #4796.The three scheduler tests omitted the backend executable identity required by production admission. The repaired fixtures supply that identity consistently while preserving schedule, locking, exact ledger, reacquisition and zero-call assertions. The added regression proves unconfigured-backend refusal, diagnostic deduplication and same-occurrence recovery with no backend effect.
6957909fffe5d4ee01520cc2fb20ca6faa978d2d— MCP timeout classificationThe #17446 MCP log shows that rename preview entered a real source read before expiry, yet the inner dispatch timeout became
kind=unavailable,terminality=pre_admission, with no cancellation stage. The generic route mapper lost the admitted read's terminal classification when the inner timer won.graph_tool_owner.rstranslates raw dispatch timeout/cancellation at the admitted read-owner boundary intotimed_out/cancelled,during_read,admitted_terminal. Authoritative problem records/details, owner side effects and other refusals retain their existing handling. Three deterministic regressions cover rendered kind/stage, record/detail precedence, and effect/other-error isolation. Existing paused-source end-to-end assertions and deadlines remain unchanged. Import-only follow-up:353eb6c8820c34f4999a6a0342436729ddd9fdf7.Current CI findings (November 8–9, 2026 UTC)
The feedback shutdown review prompted a comment correction. Each project starts its terminal write before any feedback joins are awaited. The daemon's phase coordinator stops waiting at the deadline and reports unfinished owners. The registry retains the drain task and storage ownership until the workers complete. The observability flush receives the first caller's deadline directly. These are the existing runtime semantics.
The current Bazel CI is on
fce3ebf9. No native or whole-workload benchmark result is claimed for this comment-only follow-up.New non-force merge commit:
56afe563adds currentmaster83694e0as second parent of the prior PR head5da6499, retaining the PR's existing exact source tree. The only upstream delta was the merged scheduler fixture injobs.rs, and its blob7342786381bcaf3c08454d2dc2711a6c731e7151was identical on master and this branch before merging. Updated with exact old head andforce=false. Result: zero commits behind master, six changed files instead of seven; no source fix or tests discarded and no protected model/release work imported.Prior merge-head Bazel CI: #17719, queued on
56afe563when inspected. No passed tests or benchmarks claimed for this merged head yet.Pre-merge exact-head CI (separate evidence): #17597,
5da6499, actual downloaded Linux test artifact, 181/183 Bazel targets passed, two failed. Relevant Rust test binaries were non-vacuous://crates/tracedecay-store-runtime:unit_test177 passed, 0 failed, 2 ignored, including all five issue proof regressions;//crates/tracedecay:unit_test800 passed, 0 failed, includingsession_refresh_is_cancelled_even_if_shutdown_joins_are_dropped;//crates/tracedecay:daemon_suite36 passed, 0 failed, explicitly including the previously failingcapacity_retirement_behind_a_parked_bind_eviction_cleanup_completes;//crates/tracedecay:mcp_suite675 passed, 0 failed. This confirms the prior daemon lease failure did not recur on this head, without attributing its entire cause conclusively.Two new unrelated Linux failures are real blockers, not waived:
//crates/tracedecay:session_suiteTIMED OUT at 300 s, with 377 named tests already passed andgit_backfill::large_catch_up_converges_in_one_pass_and_then_stays_settledamong those 377 passes (it completed after the 60-second progress notice);//crates/tracedecay-automation-runtime:unit_test435 passed, 2 failed on 5-second waits inretained_pair_attempts_second_leg_and_keeps_both_guards_until_both_finishandretained_user_job_rebinds_and_recovery_retires_only_terminal_corrupt_spool. Neither test file is in this PR diff. On the prior353eb6cartifact,session_suitepassed 422/422 in 74.85 s and automation-runtime passed 437/437 in 79.75 s; current core/MCP suites also ran much longer (760/813 s versus 368/220 s), compatible with fleet contention, but contention/flakiness is not proven. No test timeout or assertion was changed. Repository gates, format, Clippy, Dashboard, feature gates and harness checks passed; native macOS skipped, Windows build was cancelled before completion and native Windows tests skipped.Independent review: one COMMENTED external automated review, no approving human review, no inline review threads. Fresh real-checkout before/after time and peak-RSS measurements are not available; old #4523 figures cannot substitute. Keep
Refs #4507, issue open, and merge hold intact.Validation — evidence boundaries
Historical merged head:
56afe563fea46c03e9541697477a592c6fdc7829previously targetedmasterat83694e0; it is not the current stacked PR head. See the current integration/validation hold above.Previous-head hosted Bazel CI: run #17597 finished red on two separately identified Rust targets; the original daemon regression and issue proof regressions passed. New merged-head run #17719 is queued, not green.
Predecessor
353eb6c— not new-head evidence: run #17542, Linux-fleet job, downloaded test artifact. Linux reported 182 passed/183 test targets. Store-runtime unit tests: 177 passed, 0 failed, 2 ignored, 0 filtered, including five sealed-proof regressions. Tracedecay unit tests: 799 passed, including the three mapping tests. MCP: 675 passed. Automation runner: 171 passed. Repository gates, Clippy, Dashboard, feature gates and harness checks passed. Daemon suite: 35 passed, 1 failed, as described above. Earlier reporting that the failing file is unchanged on master does not establish harmlessness or waive the failure.Earlier
1e530bf: #17446, artifact 13619364427: 177 store-runtime tests passed; overall Linux failed the subsequently repaired scheduler and MCP tests. Windows build/type-check passed but native tests were skipped; macOS was skipped.Reused scheduler repair provenance: #4796 run 42489433681, artifact 13624506517: 171 automation-runner tests passed. The tested
43db8dbjobs.rs blob was byte-identical to the transplanted repair.The earlier hosted review environment had no local Bazel and did not dispatch the optional native workflow. Skipped native tests supply no platform evidence.
Remaining acceptance
The exact merged-head Linux fleet passed. Independent approving review, native macOS/Windows tests when available, and fresh before/after real-workload full-pass wall time and peak RSS remain required. Historical #4523 measurements (source proof 3.0–301 s down to 0.10–0.12 s; overall warm pass ~26 s/~6.7 GB) do not measure this head. Suite runtime is not a one-file-edit benchmark.
Sweeps: coordinate on this existing PR, preserve the merge hold, and leave #4507 open until acceptance is proven on master. All Kev/decision-model work, including mixed #4776, remains draft-only and must never merge/auto-merge. Release #4763 remains read-only. No release, tag, production, credential, infrastructure or external-delegation operation is included.