Repository navigation
perf(graph): avoid duplicate pathname stat and time segment opens - #4796
ScriptedAlchemy wants to merge 46 commits into
Conversation
|
Recheck cancellation and deadlines after segment reads, including errors, clear late bytes, and preserve typed interruption precedence. Add real-file regressions for cancellation, deadline expiry, truncation, and fresh-pass recovery without changing handle bounds or publication budgets. Refs #4532
|
Published source/test follow-up ed860ca. Refs #4532. Defect repaired: the pass reader checked interruption only before segment I/O. Endpoint-scanned metadata/open/read work can outlive that probe; a completed read could then return bytes after expiry, while failed I/O could be treated as source corruption. The reader now rechecks the caller's existing probe after the operation, including its error path, clears late output, and retains the original Cancelled/DeadlineExceeded result at the pass boundary. This cannot preempt an OS call already blocked in the kernel and is not a claim that cold-publication stalls are solved. Falsifiable regressions added:
Both target the previous missing post-I/O check. They are written, not yet executed: no passing/nonzero Rust test result is claimed. Existing between-read cancellation fixtures were adjusted to still interrupt between reads with the added post-I/O probe; their byte, error and open-count assertions remain intact. The GitHub update was nonforce, expected-head checked against baab1cb. Only Validation: the exact commit diff was inspected. New-head hosted CI is https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443095332 (queued when checked). Native coverage is still For the issue/sweep owners: retain this PR as the #4532 lane; coordinated distinct-page packed storage remains #4533. Keep #4532 open until hosted/nonzero/native validation, same-host publication/open-count measurements and all acceptance are proven on master. No merge, release or Kev/decision-model change. |
Reuse the short-read-safe segment range reader for whole-file evidence verification, and reject stale bytes after missing or mismatched paths. Add real-filesystem recovery and refusal regression coverage. Refs #4532
Record File::open attempts at the verified open boundary, excluding rejected path metadata from endpoint-scanned publication telemetry. Regress invalid file types and mismatched extents without changing durability checks. Refs #4532
|
Refs #4532 — published ffc7ef6 on the existing PR branch using an expected-head, nonforce update. The graph pass's Gate status: previous head |
|
2026-11-08 current-head CI triage for Hosted run 42451186352 has now passed Repository gates (Bazel formatting), Feature gates, Clippy, Dashboard and the focused The targeted store-runtime unit-test success is meaningful hosted Bazel evidence, but it is not independent code review, the same-host Defender/CorpLink before/after graph-publication measurement, nor proof of the cold O(distinct pages) first-open reduction. That format cutover remains coordinated under #4533/#4792. Leave #4796 and #4532 open; no merge or auto-merge. No speculative push while platform CI is pending. |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
Published Concrete CI repairThe three scheduler failures in the Linux fleet log are in
Their fixtures passed no backend executable, so the correct production A new integration regression exercises the actual prefilter with an unconfigured backend, proves its refusal and diagnostic deduplication, then binds an identity and proves that the same due occurrence becomes admissible without cooldown, backend execution or an effectful terminal record. Verification and remaining workThe published diff is limited to Correction to the earlier CI summary: the other failure was Coordination: the deterministic scheduler-fixture repair is now on this PR; no duplicate repair branch is needed. #4532 remains open and partial, including the coordinated #4533 storage work. No merge, release, infrastructure or protected Kev/decision-model action was taken. |
The sealed proof reader checked interruption only before component I/O. If an endpoint-scanned read failed after cancellation or deadline expiry, it could report corruption instead of the observed interruption. Clear stale output before each proof read, recheck after success or failure, and retain the typed interruption without publishing a partial proof. A real partitioned-seal regression corrupts the opened range during the read, requires the late cancellation/deadline to win with a positive open, then restores the disposable fixture and proves successful verification. Refs #4532
Published segment fixtures may be read-only. Make only the disposable fixture writable before intentionally truncating it during an opened range read, so the late-interruption regression exercises the read and not a permission error. Preserve the original expected outcomes. Refs #4532
The real sealed-evidence interruption test calls the bundle verifier, but its test module omitted that private helper import. Hosted Bazel Clippy E0425 identified two unresolved calls. Restore the exact import without touching graph logic, test assertions, or CI settings. Refs #4532
Match the hosted Bazel Rust 2024 rustfmt output in the existing proof regression module. No behavior or assertions change.
|
Refs #4532 — current-head hosted Bazel verification / new CI blocker (2026-11-08). At exact PR head New red target: At inspection, separate Windows tests were still running, the split Linux job queued, and default native macOS skipped. The only review is COMMENTED (no independent approval). Same-host Defender/CorpLink before/after OS-level |
Only pin a cold graph segment after its first authenticated range read succeeds. A post-open truncation/error must not evict healthy cached handles and force additional endpoint-scanned pathname opens. Add a real-file regression with a nonempty eight-handle cache, a post-open truncated candidate, exact refusal and cleared output, zero healthy-handle reopens, and successful recovery after restoration. Refs #4532
If the canonical seal is corrupt, a retained-copy read can outlive an endpoint-scanned deadline. Surface the fallback cancellation/deadline instead of masking it with earlier canonical corruption. Add a real-file regression proving successful retained recovery and late typed interruption after both source reads, with nonzero probes. Refs #4532
Make the successful real-file control use an exact byte-slice comparison before the interruption cases. Refs #4532
|
Refs #4532 — current-head Bazel formatting follow-up (2026-11-09 UTC). At head |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Check cancellation after the real seal manifest read, including failed I/O, before digest corruption can hide a newly expired deadline. Add a nonempty real-filesystem regression and retain the fallback's post-retained-read interruption guarantee. Refs #4532
Preserve all three issue-specific source/test blobs while adopting current master SQLite and Codex fixes. Refs #4532.
Consume the bounded prefix and remaining manifest through the same checked SHA-256 reader, then pass those authenticated bytes to component verification instead of seeking and rereading a mutable inode. Retain bounded streaming for unmaterialized revisions, typed interruption precedence, file identity checks and the canonical byte limit. Extend the real-file short-read test with an exact byte-read count and verify large materialized and unretained tails reject corruption and recover. Refs #4532
Apply the exact rustfmt hunk from hosted Bazel CI 42725393692 without changing the real-file authenticated-stream regression assertions. Refs #4532
Recheck caller interruption immediately after sealed-source pathname metadata, File::open and handle metadata, including failed OS calls. A real directory with nonempty contents and a missing seal now prove late cancellation/deadline takes precedence over metadata classification. Keep the existing nonzero mutation and read-deadline tests interrupting after the first actual file chunk despite the new admission probes. Refs #4532
A graph build can finish its last authenticated page read within the caller deadline and then exceed that deadline during projection or row spilling. Recheck the caller's typed interruption after the build before publishing success or classifying an unrelated source error. A nonvacuous real-file regression requires exactly one authenticated segment open, correct nonempty page bytes, and a third post-build probe that returns Cancelled or DeadlineExceeded. The prior code incorrectly returned success from both cases. Refs #4532
Endpoint-scanned file and pathname validation after the last checked read could expire the caller budget yet return a verified seal or misclassify cancellation as corruption/supersession. Check the typed caller probe after final handle metadata, path revalidation and Windows handle identity, including failed I/O. A real nonempty file regression reads exact authenticated bytes before injecting cancellation after final metadata and deadline after an observed pathname retirement. The old finalizer returns success or supersession without a post-I/O check. Refs #4532
Apply the two exact formatting hunks reported by hosted repository gates at the previous commit. No production logic, assertions or budgets change. Refs #4532
Bring the nonoverlapping master CI and runtime updates into the issue branch. Preserve the three existing PR file blobs byte-for-byte. The merged tree differs from master only in those three paths. Refs #4532
Layered publication passed File::take directly to serde_json's unbuffered reader, causing file reads to scale with header bytes. Buffer the bounded header in 64-KiB chunks without reading into page payloads. Add a tracing span for this production decode path. Add a real 256-page attachment regression requiring bounded actual file read calls, exact header extent, truncation refusal, a positive next-page read control, and byte-exact recovery of every page. Preserve existing page digest checks and decode-allocation admission. Refs #4532
Refs #4532 — partial source repair; HOLD MERGE
Published head:
32a10ea791a0775c96829aec9d1bd727890b3289, targetingmasteron the existing issue branch. No duplicate PR, history rewrite, auto-merge or issue closure. Full #4532 acceptance is not established.New performance repair: buffered attachment headers
Layered publication opens its parent graph through
FileCodeGraphPageStoreV1. The attachment header was passed asFile::take(header_size)directly toserde_json::from_reader, whose reader parser does not buffer input. That causes underlying file reads to scale with header bytes rather than chunks.decode_attachment_headernow wraps that already length-bounded reader in a fixed 64-KiBBufReader. TheTakeremains inside the buffer, so refills cannot consume page payloads beyond the declared header. A productioncode_index.graph.page_store.decode_headertracing span makes this path measurable. Page digest authentication before decode allocation, descriptor/extent checks, revision handling and serialized bytes are unchanged.The new real-filesystem regression writes 256 distinct nonempty pages through the production attachment writer, requires a header larger than one buffer, counts actual underlying
File::readcalls, requires exact header bytes and stream position, and bounds calls by chunk count. It also performs a positive next-page read control, rejects a truncated header, reopens through the production store and verifies every page byte-for-byte. The unbuffered implementation fails the read-count bound. This is a committed regression, not an executed result or managed-endpoint syscall benchmark.The new commit changes only
crates/tracedecay-code-index/src/production/graph_page_store.rs. Its exact parent isafd72c23ba00faa60ac5ba7b2696393a7837161b; the original three PR files and all peer history are unchanged. It was published with that expected head andforce=false. The PR now has four changed files. Related #4792 currently does not modify this file.Existing repairs retained
Verification — exact head matters
Verified at exact head
32a10ea(2026-11-10): hosted Bazel CI #42763038217 has passed all 183/183 Linux fleet targets (92 executed, 91 cached). Both//crates/tracedecay-code-index:unit_test(32.1 s) and//crates/tracedecay-store-runtime:unit_test(25.4 s) ran noncached and passed, covering the new buffered-header regression and existing graph-manifest tests;//crates/tracedecay:mcp_suitealso ran noncached and passed. Repository gates (including Bazel formatting), Clippy, Dashboard, feature gates, change scope and benchmark self-tests passed. This confirms the current changes compile and pass hosted Linux tests, not full #4532 acceptance. Windows and a separately queued Linux split were still pending at this review; native macOS was skipped. No local Rust execution or Cargo.Required owning targets include:
bazel test //crates/tracedecay-code-index:unit_test //crates/tracedecay-store-runtime:unit_test --test_output=errorsInspect nonzero execution, the attachment regression and existing graph row/digest/retirement/interruption tests, plus formatting, Clippy and relevant hosted platform checks. The ordinary workflow selects
native=none; native managed-macOS coverage and independent approving review remain outstanding. Older green runs are predecessor-only evidence; all passing claims above refer specifically to32a10ea.Remaining full-issue acceptance and coordination
This fixes another graph-publication read-amplification path, not O(distinct graph-page segments) cold first-opens. The packed/offset-indexed storage cutover or equivalent bound remains coordinated with #4533 / #4792. Do not duplicate that work or claim it complete. Require same-host before/after OS-level opens, graph-publication wall time, and successful graph readiness/serving within 15 minutes on the original managed macOS Defender/CorpLink repository, preserving graph correctness and proving acceptance on
master.Keep
Refs #4532; do not merge/auto-merge or close the issue without the required evidence. All Kev/decision-model work, including mixed #4776, remains draft/unmerged; release #4763 is read-only. No release, tag, production, credentials, infrastructure or protection changes. Prior detailed evidence is retained in issue/PR discussion and immutable commit/CI history.