Repository navigation
perf(graph): avoid duplicate pathname stat and time segment opens - #4796
ScriptedAlchemy wants to merge 50 commits into
Conversation
|
Recheck cancellation and deadlines after segment reads, including errors, clear late bytes, and preserve typed interruption precedence. Add real-file regressions for cancellation, deadline expiry, truncation, and fresh-pass recovery without changing handle bounds or publication budgets. Refs #4532
|
Published source/test follow-up ed860ca. Refs #4532. Defect repaired: the pass reader checked interruption only before segment I/O. Endpoint-scanned metadata/open/read work can outlive that probe; a completed read could then return bytes after expiry, while failed I/O could be treated as source corruption. The reader now rechecks the caller's existing probe after the operation, including its error path, clears late output, and retains the original Cancelled/DeadlineExceeded result at the pass boundary. This cannot preempt an OS call already blocked in the kernel and is not a claim that cold-publication stalls are solved. Falsifiable regressions added:
Both target the previous missing post-I/O check. They are written, not yet executed: no passing/nonzero Rust test result is claimed. Existing between-read cancellation fixtures were adjusted to still interrupt between reads with the added post-I/O probe; their byte, error and open-count assertions remain intact. The GitHub update was nonforce, expected-head checked against baab1cb. Only Validation: the exact commit diff was inspected. New-head hosted CI is https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443095332 (queued when checked). Native coverage is still For the issue/sweep owners: retain this PR as the #4532 lane; coordinated distinct-page packed storage remains #4533. Keep #4532 open until hosted/nonzero/native validation, same-host publication/open-count measurements and all acceptance are proven on master. No merge, release or Kev/decision-model change. |
Reuse the short-read-safe segment range reader for whole-file evidence verification, and reject stale bytes after missing or mismatched paths. Add real-filesystem recovery and refusal regression coverage. Refs #4532
Record File::open attempts at the verified open boundary, excluding rejected path metadata from endpoint-scanned publication telemetry. Regress invalid file types and mismatched extents without changing durability checks. Refs #4532
|
Refs #4532 — published ffc7ef6 on the existing PR branch using an expected-head, nonforce update. The graph pass's Gate status: previous head |
|
2026-11-08 current-head CI triage for Hosted run 42451186352 has now passed Repository gates (Bazel formatting), Feature gates, Clippy, Dashboard and the focused The targeted store-runtime unit-test success is meaningful hosted Bazel evidence, but it is not independent code review, the same-host Defender/CorpLink before/after graph-publication measurement, nor proof of the cold O(distinct pages) first-open reduction. That format cutover remains coordinated under #4533/#4792. Leave #4796 and #4532 open; no merge or auto-merge. No speculative push while platform CI is pending. |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
Published Concrete CI repairThe three scheduler failures in the Linux fleet log are in
Their fixtures passed no backend executable, so the correct production A new integration regression exercises the actual prefilter with an unconfigured backend, proves its refusal and diagnostic deduplication, then binds an identity and proves that the same due occurrence becomes admissible without cooldown, backend execution or an effectful terminal record. Verification and remaining workThe published diff is limited to Correction to the earlier CI summary: the other failure was Coordination: the deterministic scheduler-fixture repair is now on this PR; no duplicate repair branch is needed. #4532 remains open and partial, including the coordinated #4533 storage work. No merge, release, infrastructure or protected Kev/decision-model action was taken. |
The sealed proof reader checked interruption only before component I/O. If an endpoint-scanned read failed after cancellation or deadline expiry, it could report corruption instead of the observed interruption. Clear stale output before each proof read, recheck after success or failure, and retain the typed interruption without publishing a partial proof. A real partitioned-seal regression corrupts the opened range during the read, requires the late cancellation/deadline to win with a positive open, then restores the disposable fixture and proves successful verification. Refs #4532
Published segment fixtures may be read-only. Make only the disposable fixture writable before intentionally truncating it during an opened range read, so the late-interruption regression exercises the read and not a permission error. Preserve the original expected outcomes. Refs #4532
The real sealed-evidence interruption test calls the bundle verifier, but its test module omitted that private helper import. Hosted Bazel Clippy E0425 identified two unresolved calls. Restore the exact import without touching graph logic, test assertions, or CI settings. Refs #4532
Match the hosted Bazel Rust 2024 rustfmt output in the existing proof regression module. No behavior or assertions change.
|
Refs #4532 — current-head hosted Bazel verification / new CI blocker (2026-11-08). At exact PR head New red target: At inspection, separate Windows tests were still running, the split Linux job queued, and default native macOS skipped. The only review is COMMENTED (no independent approval). Same-host Defender/CorpLink before/after OS-level |
Only pin a cold graph segment after its first authenticated range read succeeds. A post-open truncation/error must not evict healthy cached handles and force additional endpoint-scanned pathname opens. Add a real-file regression with a nonempty eight-handle cache, a post-open truncated candidate, exact refusal and cleared output, zero healthy-handle reopens, and successful recovery after restoration. Refs #4532
|
Refs #4532 — current-head Bazel formatting follow-up (2026-11-09 UTC). At head |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Check cancellation after the real seal manifest read, including failed I/O, before digest corruption can hide a newly expired deadline. Add a nonempty real-filesystem regression and retain the fallback's post-retained-read interruption guarantee. Refs #4532
Preserve all three issue-specific source/test blobs while adopting current master SQLite and Codex fixes. Refs #4532.
Consume the bounded prefix and remaining manifest through the same checked SHA-256 reader, then pass those authenticated bytes to component verification instead of seeking and rereading a mutable inode. Retain bounded streaming for unmaterialized revisions, typed interruption precedence, file identity checks and the canonical byte limit. Extend the real-file short-read test with an exact byte-read count and verify large materialized and unretained tails reject corruption and recover. Refs #4532
Apply the exact rustfmt hunk from hosted Bazel CI 42725393692 without changing the real-file authenticated-stream regression assertions. Refs #4532
Recheck caller interruption immediately after sealed-source pathname metadata, File::open and handle metadata, including failed OS calls. A real directory with nonempty contents and a missing seal now prove late cancellation/deadline takes precedence over metadata classification. Keep the existing nonzero mutation and read-deadline tests interrupting after the first actual file chunk despite the new admission probes. Refs #4532
A graph build can finish its last authenticated page read within the caller deadline and then exceed that deadline during projection or row spilling. Recheck the caller's typed interruption after the build before publishing success or classifying an unrelated source error. A nonvacuous real-file regression requires exactly one authenticated segment open, correct nonempty page bytes, and a third post-build probe that returns Cancelled or DeadlineExceeded. The prior code incorrectly returned success from both cases. Refs #4532
Endpoint-scanned file and pathname validation after the last checked read could expire the caller budget yet return a verified seal or misclassify cancellation as corruption/supersession. Check the typed caller probe after final handle metadata, path revalidation and Windows handle identity, including failed I/O. A real nonempty file regression reads exact authenticated bytes before injecting cancellation after final metadata and deadline after an observed pathname retirement. The old finalizer returns success or supersession without a post-I/O check. Refs #4532
Apply the two exact formatting hunks reported by hosted repository gates at the previous commit. No production logic, assertions or budgets change. Refs #4532
Bring the nonoverlapping master CI and runtime updates into the issue branch. Preserve the three existing PR file blobs byte-for-byte. The merged tree differs from master only in those three paths. Refs #4532
Layered publication passed File::take directly to serde_json's unbuffered reader, causing file reads to scale with header bytes. Buffer the bounded header in 64-KiB chunks without reading into page payloads. Add a tracing span for this production decode path. Add a real 256-page attachment regression requiring bounded actual file read calls, exact header extent, truncation refusal, a positive next-page read control, and byte-exact recovery of every page. Preserve existing page digest checks and decode-allocation admission. Refs #4532
Refs #4532. Recheck interruption after the real seal pathname metadata operation, including failed I/O, before returning corruption or seal-superseded. Add a nonzero filesystem and deadline regression.
|
Refs #4532 — exact-head hosted verification (2026-11-10 UTC), not merge/issue acceptance. At PR head The Windows job is still in progress, building/typechecking; the default native macOS job was skipped. There is no independent approving review. Current master The original O(distinct graph-page segments) cold pathname opens remain, requiring the coordinated #4533 packed/offset read solution rather than duplicate ownership here. Same Defender/CorpLink-managed macOS repository before/after OS-level opens, full publication wall time and correct graph serving within 15 minutes on master remain unproven. HOLD MERGE, keep #4532 open and |
Refs #4532 Windows seal validation opens a second pathname handle after the last deadline probe. Recheck the caller after that potentially slow I/O before reporting corruption or accepting a seal. Cover the real file admission boundary with a Windows-only late-deadline regression.
Verify the exact page buffer before JSON decode instead of performing an extra streaming hash pass and then rereading each attachment page. Keep digest error classification, page identity and byte limits. Exercise real file bytes and a corrupt-page refusal with counted I/O. Refs #4532
November 10, 2026 — single-pass layered page read (PARTIAL / HOLD MERGE)
Published
50dfcfcdirectly onto this existing PR, exact previous head7d3d4791, via nonforce expected-head branch update. One source/test file changed:crates/tracedecay-code-index/src/production/graph_page_store.rs.New performance finding: The layered-attachment
FileCodeGraphPageStoreV1::read_pagehashed the entire attachment page inverify_attachment_page, sought back, and read the same bytes a second time for JSON decoding. That doubles page payload filesystem I/O even though both checks operate on the same held File. The production reader now reads the page once, validates its exact extent and SHA-256 before JSON decoding, and preserves the originalpage verify: digest does not matchrefusal. The redundant stream pass and reread helper were deleted, not retained as a compatibility path.A committed filesystem-backed regression writes a genuine attachment, runs the production-owned one-pass reader through a counted real
File, requires exactly one page's bytes read and normal store parity, then corrupts the actual page without changing its length and requires digest refusal before decode with exactly one page read. Existing header boundary, corruption and full-page roundtrip regressions remain.Verification boundary: The exact commit and one-file diff were reread from GitHub after publication. Current-head hosted Bazel CI 42940202413 was queued; no Rust/Bazel/rustfmt/Clippy/native test pass for this SHA is yet claimed. Local checkout/Bazel is unavailable; no Cargo was run. Previously passing Linux CI belongs to the prior head and does not validate this new test.
Full #4532 acceptance is still OPEN: this removes duplicated layered-attachment payload reads, not the remaining O(distinct graph segments) cold pathname opens in the eight-handle pass cache. The packed/offset-based cutover is coordinated with #4533/#4792, not duplicated here. Also missing: independent approval, nonzero current-head hosted tests, same Defender/CorpLink-managed macOS original-repository before/after OS-level
open()counts and publication wall time, a correct serving graph inside 15 minutes, and proof onmaster. KeepRefs #4532, no merge/auto-merge or issue closure. All Kev/decision-model work including draft #4776 remains protected; release #4763 read-only.2026-11-10 — exact-head Bazel formatting repair; still HOLD MERGE
Current published head:
e8b0af8, a one-file, expected-head, nonforce fast-forward from47ad1a1. The only change is the exact rustfmt layout in the real-segment retirement/deadline regression; no production behavior, assertion or budget changed. Refs #4532.Prior-head hosted Bazel CI #42901690800 exposed one exact
//crates/tracedecay-store-runtime:rustfmt_check_2024hunk. That run passed Clippy, Dashboard and both relevant code-index/store-runtime Rust tests (store-runtime noncached, 65.7 s); the Linux fleet overall failed because the unrelated//crates/tracedecay-global-db:unit_testtimed out at 300.2 s (182/183 targets passed). The repository gates failed formatting; the formatting diff is now committed. The timeout is not established to be a graph regression and has not been hidden or waived.New exact-head CI #42914461736 is queued; do not use predecessor checks as current-head approval. Native macOS was skipped in the preceding PR workflow; no same-host Defender/CorpLink performance measurements or independent approving review yet. This PR remains PARTIAL / HOLD MERGE: cold first-opens are still O(distinct segments), with the packed/offset-storage work coordinated separately in #4533/#4792. Do not merge, auto-merge or close #4532 without independent review and correct graph serving under 15 minutes on the managed host and proof on
master. Keep Kev/decision-model (including #4776) draft/unmerged and release #4763 read-only.November 10, 2026 — retirement-probe deadline repair (PARTIAL / HOLD MERGE)
New published head:
47ad1a1, appended directly to the existing issue PR with exact parent32a10ea791a0775c96829aec9d1bd727890b3289, expected-head guard andforce=false. Refs #4532. Onlycrates/tracedecay-store-runtime/src/session_registry/code_graph_manifest.rschanged; peer commits retained.Source defect fixed: After graph-row failure and the post-projection budget probe, a final
seal_still_namedfilesystem metadata lookup could itself stall under endpoint security and consume the deadline. That path previously returned corruption orSealSupersededwithout rechecking cancellation/deadline. It now probes immediately after the real seal-path I/O (including a metadata error) and preserves typed interruption over stale graph-source classification.Falsifiable real-filesystem regression: Read an actual nonempty, digest-identified graph segment through the production pass, then exercise both a named and retired seal with uninterrupted corruption/retirement controls. For each, a late
CancelledorDeadlineExceededvisible on the fourth callback must override the old classification; require exactly four probes, one actual segment open and the original exact page bytes. The predecessor returns the wrong error; the new regression is committed but not yet claimed Bazel-passing.Hosted validation: exact-head CI #42901690800 started; repository gates, Clippy and Linux fleet were in progress at publication. Standard native macOS job is skipped. Earlier CI #42763038217 passed 183/183 Linux Bazel targets at older
32a10ea, not at this commit. No local Rust execution or Cargo.Acceptance remains on hold: This closes an additional endpoint-stall error-classification gap, not the O(distinct-segment) cold first-open architecture. Coordinate packing/offset indexing with #4533/#4792, not duplicate it. Still require independent approval; exact-head nonzero Bazel/CI; native managed macOS when available; and before/after OS-level
open()counts, full publication time, graph correctness and readiness within 15 minutes on the same Defender/CorpLink-managed host, proven onmaster. Do not merge/auto-merge/close #4532. Protected Kev/decision-model work including #4776 stays draft; release #4763 is read-only.Refs #4532 — partial source repair; HOLD MERGE
Published head:
32a10ea791a0775c96829aec9d1bd727890b3289, targetingmasteron the existing issue branch. No duplicate PR, history rewrite, auto-merge or issue closure. Full #4532 acceptance is not established.New performance repair: buffered attachment headers
Layered publication opens its parent graph through
FileCodeGraphPageStoreV1. The attachment header was passed asFile::take(header_size)directly toserde_json::from_reader, whose reader parser does not buffer input. That causes underlying file reads to scale with header bytes rather than chunks.decode_attachment_headernow wraps that already length-bounded reader in a fixed 64-KiBBufReader. TheTakeremains inside the buffer, so refills cannot consume page payloads beyond the declared header. A productioncode_index.graph.page_store.decode_headertracing span makes this path measurable. Page digest authentication before decode allocation, descriptor/extent checks, revision handling and serialized bytes are unchanged.The new real-filesystem regression writes 256 distinct nonempty pages through the production attachment writer, requires a header larger than one buffer, counts actual underlying
File::readcalls, requires exact header bytes and stream position, and bounds calls by chunk count. It also performs a positive next-page read control, rejects a truncated header, reopens through the production store and verifies every page byte-for-byte. The unbuffered implementation fails the read-count bound. This is a committed regression, not an executed result or managed-endpoint syscall benchmark.The new commit changes only
crates/tracedecay-code-index/src/production/graph_page_store.rs. Its exact parent isafd72c23ba00faa60ac5ba7b2696393a7837161b; the original three PR files and all peer history are unchanged. It was published with that expected head andforce=false. The PR now has four changed files. Related #4792 currently does not modify this file.Existing repairs retained
Verification — exact head matters
Verified at exact head
32a10ea(2026-11-10): hosted Bazel CI #42763038217 has passed all 183/183 Linux fleet targets (92 executed, 91 cached). Both//crates/tracedecay-code-index:unit_test(32.1 s) and//crates/tracedecay-store-runtime:unit_test(25.4 s) ran noncached and passed, covering the new buffered-header regression and existing graph-manifest tests;//crates/tracedecay:mcp_suitealso ran noncached and passed. Repository gates (including Bazel formatting), Clippy, Dashboard, feature gates, change scope and benchmark self-tests passed. This confirms the current changes compile and pass hosted Linux tests, not full #4532 acceptance. Windows and a separately queued Linux split were still pending at this review; native macOS was skipped. No local Rust execution or Cargo.Required owning targets include:
bazel test //crates/tracedecay-code-index:unit_test //crates/tracedecay-store-runtime:unit_test --test_output=errorsInspect nonzero execution, the attachment regression and existing graph row/digest/retirement/interruption tests, plus formatting, Clippy and relevant hosted platform checks. The ordinary workflow selects
native=none; native managed-macOS coverage and independent approving review remain outstanding. Older green runs are predecessor-only evidence; all passing claims above refer specifically to32a10ea.Remaining full-issue acceptance and coordination
This fixes another graph-publication read-amplification path, not O(distinct graph-page segments) cold first-opens. The packed/offset-indexed storage cutover or equivalent bound remains coordinated with #4533 / #4792. Do not duplicate that work or claim it complete. Require same-host before/after OS-level opens, graph-publication wall time, and successful graph readiness/serving within 15 minutes on the original managed macOS Defender/CorpLink repository, preserving graph correctness and proving acceptance on
master.Keep
Refs #4532; do not merge/auto-merge or close the issue without the required evidence. All Kev/decision-model work, including mixed #4776, remains draft/unmerged; release #4763 is read-only. No release, tag, production, credentials, infrastructure or protection changes. Prior detailed evidence is retained in issue/PR discussion and immutable commit/CI history.