code-index: graph publication opens a segment file per page read and never finishes under endpoint scanning

Observed

Integration run of master b14d020 (release build, isolated profile, fresh detached clone of this repository, macOS with Microsoft Defender + CorpLink endpoint agents):

Graph publication hit its 15-minute deadline on all three attempts (graph publication probe deadline exceeded, then code_index_graph_publication_budget_spent); the graph never served and every graph tool answered code-graph-refused. tracedecay init --wait ended rc=1 with code_graph_refused: the sealed code graph publication exceeded its background budget.
The daemon sat at ~3% CPU during publication. sample <pid> 8: 4,576 of 5,012 samples were inside open() via build_sealed_code_graph_rows -> SealedCodeGraphPageStoreV1::read_page -> read_partitioned_segment: one open() per page read.
On this machine a plain open() costs 7-100 ms (endpoint scanning; load average 33-56).

Why it matters

Graph publication latency scales with (pages read) x (cost of one file open). Endpoint security agents are common on developer machines; with them the graph can never become ready, and every graph tool is refused.

Expected

Graph publication opens each segment file once per pass (or holds a bounded cache of open segment handles / maps the segment), so publication cost does not multiply per-file-open latency by the page count. Measure publication time and open() count before/after on the same repository.

Related: #4507, #4520 (incremental cost), #4523 (verification cache).

Issue-specific follow-up published: #4796 (fix/4532-graph-open-cost-telemetry, 737a3504) against master, Refs #4532. It eliminates the duplicate pathname symlink_metadata() call on every cold graph segment open, carries identity into the verified handle open, and aggregates segment-open and graph-pass wall time. A new real-filesystem regression checks exact bytes / one positive open, and refusal / zero opens for an invalid first existing route.

Coordination: #4547 and #4725 (the latter integrated into merged #4742) already address repeat opens within a pass; the real cold graph regression still establishes one unique file per page and thus O(distinct page) first opens. #4792 is separate staged-preparation work for #4533; it does not solve this cold graph path. The packed-storage/offset-index cutover remains coordinated under #4533, not duplicated here.

#4796's GitHub CI is queued; no local Bazel toolchain or managed macOS endpoint-scanning host is available in this run. Do not merge or close #4532 until nonzero Bazel/native CI, independent review, and same-host before/after full graph publication/readiness plus open-count measurements prove the original 15-minute failure resolved. This is a partial performance/diagnostic improvement, not a claimed fix for first-open stalls.

2026-11-08 follow-up: Pushed an additional reviewable source/test commit to the same #4796 branch using a nonforce update checked against its prior exact head. The graph reader now clears an output buffer before validating a new page, so a rejected range or mismatched manifest cannot expose stale bytes from the previous verified page. A new real-filesystem regression first reads exact content (one open), then asserts both error paths clear the buffer with no further opens. The prior reader fails those buffer assertions. This protects correctness but does not eliminate distinct first-opens on the cold publication path. Hosted Bazel CI for the new head is queued with native=none; tests, independent review and the original endpoint-scanned macOS measurements remain unproven. #4532 remains open.

2026-11-08 CI/correctness follow-up (PR #4796, c950f350e758): Hosted Bazel Repository gates identified two rustfmt mismatches in the new regression assertions; corrected both rather than weakening the gate. A further source+test commit closes two error-output leaks: a cancellation/deadline check failing before the pass reader ran left the previous verified page in its caller buffer, and a short/failed seek-read could expose partially filled bytes. New regressions run through the real filesystem graph pass with an interrupted second request and through the production read helper with a short standard reader; assert empty output, original interruption outcome and exactly one segment open, plus successful exact re-read. This is published, not yet verified: the new Bazel CI head has not completed; no native macOS/Windows or same-host endpoint-security time/open comparison was performed. Cold distinct-segment opens remain O(pages), pending coordinated #4533 storage work. Issue stays open.

fix(graph): preserve interruptions after segment I/O

fix(graph): clear rejected fingerprint page output

2026-11-08 follow-up: Published commit 36c47e5 directly to existing issue-specific PR #4796, targeting master, with an exact-head nonforce branch update. read_fingerprinted_segment now clears rejected output before filesystem validation and reuses the existing short-read-safe range reader; a real-filesystem regression requires exact successful bytes, refusal with empty output on manifest-size mismatch or vanished evidence, and successful re-read in between. Previously the rejected paths retained the prior verified page bytes. Refs #4532, not Fixes.

CI for this exact head is queued with native=none. A previous run for baab1cb passed Bazel repository formatting gates, but its Clippy and Linux fleet jobs failed with remote execution UNAVAILABLE: io exception (exit 34), not a confirmed source/test failure. Nonzero Rust tests for this head, native platform CI, independent review, and same-host macOS endpoint-scanned before/after publication time and OS open counts are still missing. Cold graph publication still opens O(distinct pages) content-addressed files; #4533 owns the coordinated packed-storage solution. Do not merge or close #4532 without acceptance evidence on master. No protected changes, force pushes, or release actions.

fix(graph): count only real sealed segment open attempts

2026-11-08 #4532 ownership update: Published ffc7ef6 directly to existing PR #4796 against master with an exact expected-head, nonforce update; no duplicate PR. A graph pass used to increase its segment_open_attempts counter when route metadata existed even if validation rejected a directory or incorrect manifest extent before File::open. Counter advancement now sits at the actual primary open boundary. The existing real-filesystem test gained falsifiable zero-open assertions for both refusals, while preserving its exact-content and one-open positive assertion. Scope: one source/test file, Refs #4532, no change to storage identity, concurrency limits, or resource deadlines.

GitHub new-head hosted CI is queued (native=none); preceding 36c47e5 Repository gates passed but other jobs were cancelled by supersession, so no nonzero Rust test or native-platform success is claimed. Independent review and the actual same-host 6.6k-file macOS endpoint-security publication timing, OS-level open counts, and successful under-budget graph readiness remain missing. Graph cold first-opens remain O(distinct pages); packed-storage work stays under #4533, not duplicated in this PR. Leave #4532 and #4796 open until acceptance is demonstrated on master; draft Kev/decision-model work (#4776) and release PR #4763 remain untouched.

style(graph): fix hosted Rust formatting gate

2026-11-08 #4532 follow-up — published repair to existing PR #4796, targeting master, Refs #4532. Hosted run 42447840035 for previous head ffc7ef6 uncovered a concrete Bazel //:rustfmt_check failure in code_graph_manifest.rs:2430 (one multi-line assertion). Corrected the exact rustfmt diff in commit 73d04b3 by a nonforce expected-head update on fix/4532-graph-open-cost-telemetry. Diff verified: only the affected assertion formatting changed; its zero-open refusal condition is unchanged. No new PR or rewritten history.

Superseded CI run 42450998455 was queued before the follow-up code repair; nonzero //crates/tracedecay-store-runtime:unit_test execution, independent review, and native macOS validation are not yet established. The automatic PR CI has native=none; this is not an endpoint-scanned same-host reproduction. The underlying O(distinct-page) cold first-opens still require the coordinated #4533 packed-storage cutover; current #4792 handles staged preparation/durability and is not a duplicate graph-publication fix. Keep #4532 and #4796 open until graph readiness, OS open counts, and before/after publication times are proven on the reported endpoint-secured repository and accepted on master. No release, protected draft, or unrelated changes.

Additional source/test repair published to the same PR: 2fd183c stops malformed cached graph requests from promoting a pinned segment's LRU position before its read succeeds. A previously invalid manifest-size read could change the next eviction victim and force an avoidable File::open. Extended the existing bounded real-filesystem nine-segment regression to require the rejected read clear output, retain all valid handles, preserve counter accuracy, and allow a subsequent valid read without a new open; this is falsifiable against the preceding head. Nonforce, exact expected-head update; only code_graph_manifest.rs changed. New current-head hosted Bazel CI is queued (native=none); no nonzero Rust-test pass or managed endpoint-scanned macOS measurement is claimed. Still Refs #4532 only, no closure/merge.

fix(graph): keep rejected cache reads from changing LRU order

test(automation): bind scheduler fixtures to backend identities

fix(graph): preserve late interruption during seal evidence reads

Refs #4532 — published bd9b1a6 onto the existing PR #4796 against master using an exact expected-head, nonforce fast-forward from 2ecbdf1; no duplicate PR or rewritten history.

Additional graph correctness defect: the sealed-generation proof read callback, unlike the graph-row pass, only checked cancellation/deadline before opening/reading a component. On a managed endpoint-scanned machine a failing I/O operation can outlive that check and be misclassified as source corruption. The callback now clears stale output and rechecks the caller's interruption after both successful and failed I/O; a proof interrupted this way cannot commit. The new real partitioned-seal regression first verifies a nonzero real generation, opens the evidence range, then truncates disposable fixture segment files while the range handle is open. It requires a real counted open, typed Cancelled/DeadlineExceeded outcome from exactly one late interruption check, then restoration and successful verification. The preceding code would report Corrupt after the injected read failure. All existing graph-reader fixes and scheduler CI repairs remain on the branch; no production budget, storage format, CI protection, or tests were weakened.

Current-head Bazel CI 42481424377 is queued; the new regression has not yet executed, and native macOS is skipped. Previous-head Linux Bazel passed all 183 targets (22 executed; rest cached) but cannot validate this new commit. Independent approval, current-head nonzero tests, and same-host Defender/CorpLink before/after OS open() counts, elapsed publication time and graph readiness remain outstanding. Distinct-page cold first-opens are still O(pages); coordinated packed-storage issue #4533/PR #4792 owns that cutover, and #4507/PR #4791 separately owns inherited proof-cache work. Issue and PR remain open; no merge, release #4763 action or Kev draft (#4776) action.

Test-fixture correction (same issue PR): 108b1ae explicitly makes only the disposable published segment fixtures writable before deliberate in-place truncation; persisted graph data and permission policy are untouched. This was also an exact-head nonforce append. Fresh current-head Bazel test execution is still pending, so the new assertion is not claimed passing.

test(graph): permit the sealed evidence fault on read-only files

fix(graph): import verification helper in proof regression

style(graph): satisfy hosted manifest proof formatter

Refs #4532 — published exact hosted Bazel formatting repair to the existing master-targeting PR #4796.

At preceding source/test head 1d40f5702d37138d3166be002ecd4d0efef60989, CI 42725393692 failed //crates/tracedecay-store-runtime:rustfmt_check_2024 on one exact assertion layout at code_graph_manifest.rs:2903. All eight repository policy tests passed; subsequent jobs were superseded/cancelled, not passed.

Appended commit 30f4f99 with only that exact rustfmt hunk (no test assertion or production logic change), as a nonforce expected-head fast-forward from 1d40f57; verified the remote PR head after publication. Fresh exact-head Bazel CI 42728164539 is queued; the new manifest authentication regression still needs a nonzero executed pass, Clippy and formatting proof.

Freshly inspected prior CI 42718305876 at 6bf169f: 183/183 Linux Bazel test targets passed (23 executed, rest cached), including noncached //crates/tracedecay-store-runtime:unit_test (46.1 seconds); repository gates, Clippy, Dashboard and feature gates passed. This is predecessor evidence, not validation of 1d40f57 or 30f4f99. Native macOS was skipped.

Rechecked all 22 issue comments, current root AGENTS.md and full 7,919-entry master tree (no nested AGENTS), master graph reader/storage/tests, the issue PR and all review threads, and the overlapping #4533/#4792 storage lane. No competing complete fix. Distinct graph pages still need O(distinct-segment) cold first opens; independent approving review and same-host Defender/CorpLink syscall open counts, publication time, graph readiness within 15 minutes and full acceptance on master remain unproven. Keep #4796 and #4532 open and unmerged; coordinate packed-store work under #4533. No Cargo, force push/rewrite, protected draft/decision-model, release, or infrastructure action.

fix(graph): preserve deadlines after seal open filesystem probes

style(graph): normalize sealed-open regression spacing

Refs #4532 — published an additional endpoint-scanned sealed-source deadline repair to the existing master-targeting PR #4796, not a duplicate PR.

Source defect repaired: open_checked_seal_reader classified path metadata, File::open and opened-handle metadata errors without checking whether the endpoint-security filesystem operation itself had consumed the caller's deadline. It now rechecks typed cancellation/deadline immediately after each actual OS operation, including failures, before classifying unavailable/corrupt. Earlier manifest-read I/O protections did not cover this separate checked-seal path.

Nonvacuous regression: the new real-filesystem checked_seal_open_preserves_interruption_after_metadata_io creates a directory with nonempty contents and an absent seal. With uninterrupted checks it requires the original Corrupt/Unavailable outcomes; with cancellation/deadline becoming visible immediately after the real metadata operation it requires the corresponding exact typed error and one executed post-I/O probe. The predecessor would return Corrupt/Unavailable before invoking the checker. Existing nonzero same-length mutation and deadline tests are adjusted so their faults still occur after the first completed real seal read, rather than during the extra metadata probes; no assertion, budget, or protection was weakened.

Source + test cbdf2f0 and whitespace-only dc729fe were appended through exact-head, expected-SHA nonforce fast-forwards; only crates/tracedecay-store-runtime/src/session_registry/code_graph_manifest.rs changed. Remote PR head was verified. New-head hosted Bazel CI #42736279250 is queued; no nonzero execution or passing gate is claimed on this head. Previous 30f4f99 CI #42728164539 passed Repository gates, change scope and benchmark self-tests, then was superseded before Rust/Clippy/native validation.

Rechecked the complete issue and 23 paginated comments, current root AGENTS.md and complete 7,919-path master tree (no nested AGENTS), master graph code/tests, PR review threads, hosted workflow jobs and competing/coordinated #4533/#4792/#4846 work. #4796 remains the issue PR; no complete competing fix identified. Core acceptance remains blocked by O(distinct-page) cold first-opens, independently approving review, native managed macOS verification, and same-host Defender/CorpLink before/after OS-level open counts and under-15-minute graph readiness proven on master. Leave PR and issue open; preserve #4776/Kev drafts and release #4763 read-only. No Cargo, protected branch/CI bypass, release, infrastructure, or external delegation.

fix(graph): check budget after sealed row projection

Refs #4532 — published f27944d on the existing #4796 master-targeting branch via exact expected-head, force=false update from dc729fe9. The one-file source/test diff closes a real post-projection deadline gap: the graph pass could successfully finish its last segment read, then spend the remainder of its budget computing/spilling rows and return success without rechecking check. It now checks caller interruption after the builder completes, without changing any budget or masking already-recorded typed interruptions. The new real-filesystem regression requires exact nonempty segment bytes, one positive open, exactly three pre/post/post-projection checks, and typed Cancelled/DeadlineExceeded if expiry occurs after the final read. Prior code returns success and fails this test.

New-head hosted Bazel CI 42742833730 is queued; no nonzero test/Clippy/native pass for this SHA is claimed. No local Bazel toolchain, managed Defender/CorpLink macOS host, or native workflow dispatch was available. Independent approval remains missing. The pass-scoped LRU still cold-opens O(distinct graph page segments); coordinate with #4533/#4792's held storage work, not a duplicate fix. Same-host OS-level before/after open counts, wall time and served graph within 15 minutes on master remain unproven. Keep #4796 and #4532 open/unmerged; Refs, not Fixes. All Kev/decision-model work including draft #4776 stays draft/unmerged and release #4763 remains read-only. No Cargo, force/rewrite, release, credentials or infrastructure changes.

fix(graph): preserve final seal validation deadlines

Refs #4532 — published a new review-backed source/test repair on the existing master-targeting PR #4796, not a duplicate PR: 295fede. The commit is a direct child of f27944dad47ee660c00d32a19d867a7da4e95d4f; updated the existing ref with expected_sha and force=false, then confirmed the remote PR head. Only crates/tracedecay-store-runtime/src/session_registry/code_graph_manifest.rs changed.

Source defect: checked seal replay correctly probes cancellation/deadline after the streaming read, but its final File::metadata, seal_left_path pathname revalidation, and Windows stable-handle identity check could themselves take endpoint-authorized filesystem time. A deadline crossed during that final validation could return an apparently verified seal or report corruption/retirement rather than typed Cancelled/DeadlineExceeded. The finalizer now checks the existing caller probe immediately after each operation including failed I/O, before classifying its result.

Falsifiable real-filesystem regression: checked_seal_finish_preserves_deadline_after_final_file_and_path_io reads and hashes a real nonempty seal with exact bytes/length, then requires typed cancellation at the second finalization check (after handle metadata) and typed deadline at the third (after pathname recheck). On Unix the latter case retires the actual pathname between the two checks, proving deadline precedence over SealSuperseded. The previous code invokes only the initial finalization check, so both new assertions fail there. Existing corruption, digest, segment-read, cancellation and retirement tests remain unchanged.

Verification boundary: new-head hosted Bazel CI #42749505402 was queued with native=none at inspection. No current-head nonzero Rust-test pass, formatting/Clippy pass, independent approving review, or native managed-macOS result is claimed. Prior CI #42718305876 passed 183/183 Linux Bazel targets on the older 6bf169f head, not this change. No Cargo or alternate Rust execution was used.

The central O(distinct graph page segments) first-open pathology remains after eight-handle reuse. #4533/#4792 owns the held packed/offset-index cutover; it is not a competing completed fix. Same-host Defender/CorpLink OS-level before/after open() counts, publication wall time and successful graph readiness within 15 minutes on master, native macOS, exact-head Bazel and independent review are still required. PR #4796 and #4532 remain open/unmerged; Refs, not Fixes. Kev/decision-model work including draft #4776 remains DRAFT/unmerged; release #4763 read-only. No forced update, merge, production, credentials, releases, infrastructure, external delegation, or protection changes.

style(graph): apply hosted Bazel rustfmt corrections

Refs #4532 — published exact hosted Bazel formatting repair to existing PR #4796: commit 4608c9b, direct child of 295fede7 using expected_sha=295fede7 and force=false. Verified remote ref, commit parent and one-file patch. No logic, tests or budgets were weakened.

New independently observed blocker: CI #42749505402 failed //crates/tracedecay-store-runtime:rustfmt_check_2024 in precisely two formatting hunks (final-seal error mapper and the real-directory regression setup). All eight repository policy targets passed, but no nonzero store-runtime pass on that head is claimed. Both exact hunks are now committed. New head CI #42754587820 queued with native=none; no current-head green claimed.

Rechecked complete issue and 26 paginated comments, current root AGENTS.md and 7,919-entry master tree (no nested AGENTS), current master implementation/tests, #4796 source and review threads/CI, and overlapping #4533/#4792/#4846 work. No competing completed architectural fix. The PR is 40 master commits behind and still O(distinct page segments) cold opens; refrain from speculative sync during new-head CI. Missing: independent approving review, nonzero exact-head Bazel pass, managed macOS native validation, same-host Defender/CorpLink before/after OS-level open counts and publication latency with a served graph inside 15 minutes on master. PR and issue remain open; Refs, not Fixes. No Cargo, auto-merge, protected Kev/#4776, release/#4763 writes, or infrastructure changes.

chore(graph): integrate master without changing issue patch

Refs #4532 — published a safe master integration to existing PR #4796, not a new issue fix or acceptance claim.

Exact two-parent merge afd72c23 retains previous issue head 4608c9b365075037eaefa4079c22ffe572cc072a and integrates master 2f9d65923c527f0dd52c161c7d6d1a1f079f2cb8. Remote update used expected_sha=4608c9b, force=false; remote PR head and parents reverified.
Re-read the complete current 7,920-entry master tree and current root AGENTS.md: no nested AGENTS.md paths. The 42 integrated master commits changed none of the three perf(graph): avoid duplicate pathname stat and time segment opens #4796 files. Constructed merge tree from the exact master tree + the unchanged three PR file blobs, then compared both complete trees: only those three leaf files and their parent directory hashes differ. PR now 45 ahead / 0 behind, exactly three changed files, no draft Kev, fix: consolidate follow-up application issue repairs #4776 or release branch integration.
Fresh exact-head hosted Bazel CI #42760375047 queued (native=none). The previous 4608c9b run #42754587820 remained queued ~50 minutes and was cancelled by supersession, not passed. No exact-head nonzero Rust, format or Clippy result yet. Earlier 183/183 Linux green belongs only to 6bf169f.
Rechecked issue and all 27 paginated comments, PR review/thread states (one outdated resolved inline comment, no independent approving review), master/PR graph source/tests, overlapping open code-index: store and read the index in few files so endpoint security open() cost does not dominate #4533/perf(code-index): reuse staged read and durability handles #4792 and code-index: a one-file edit re-verifies the whole sealed generation (~5 min, 2.3 GB peak) #4507/fix(code-graph): retain authenticated proofs across interrupted successors #4846 work, and the existing sweep/PR-maintenance task ownership. No competing completed solution. The graph still cold-opens O(distinct page segments); code-index: store and read the index in few files so endpoint security open() cost does not dominate #4533's packed/offset-index architecture is separate and unproven.
Still required: exact-head Bazel and native managed-macOS verification when available, independent approval, and same original Defender/CorpLink 6.6k-file host before/after OS open counts, full publication wall time and a correct serving graph within 15 minutes on master.

HOLD MERGE/ISSUE CLOSURE. This update preserves prior patches; it does not assert root-cause elimination. Keep only Refs #4532 pending full evidence, no Cargo, release/#4763 mutation, Kev/decision-model/held #4776 merge, force, infrastructure, credentials or external delegation.

perf(graph): buffer bounded attachment header reads

Published a source-level performance repair directly to existing PR #4796: 32a10ea, Refs #4532. Exact parent afd72c23ba00faa60ac5ba7b2696393a7837161b; expected-head/nonforce branch update verified. Only crates/tracedecay-code-index/src/production/graph_page_store.rs changes in this commit, preserving the previous three PR paths.

Confirmed source defect: layered graph publication opens its parent attachment with FileCodeGraphPageStoreV1, which handed a length-bounded but unbuffered File to serde_json::from_reader. Underlying reads consequently scale with JSON header bytes. The decoder now buffers 64 KiB at a time outside the already-bounded Take, preventing page payload read-ahead. A production tracing span measures header decode. Existing digest-before-decode-allocation, descriptor, revision, extent and page-byte protections are unchanged.

New falsifiable real-filesystem regression: the production writer creates 256 distinct nonempty pages and a header larger than one buffer. The test observes underlying File::read calls, requires chunk-bounded calls and exact header extent/position, positively observes the following page byte, rejects a truncated header, reopens via the production store and compares every recovered page byte-for-byte. Removing the buffer violates the read-count bound. This test is committed, not yet Bazel-executed; its counted reads are not managed-host syscall/timing measurements.

Exact-head hosted Bazel CI #42763038217 is queued with native=none. Required owning targets are //crates/tracedecay-code-index:unit_test and //crates/tracedecay-store-runtime:unit_test, plus formatting/Clippy and relevant platform checks. No local Rust execution or current-head pass is claimed.

Coordination: #4792's current changed-file list does not include this attachment decoder, and the inspected master still has the unbuffered implementation. This repair addresses layered publication reads, not the original O(distinct-segment) cold-open architecture. The #4533/#4792 storage cutover/equivalent bound, independent approval and same-managed-macOS before/after open counts and under-15-minute graph readiness remain outstanding. Keep #4532 and #4796 open and unmerged. No Cargo, protected Kev/#4776, release/#4763, infrastructure, protection or history-rewrite changes.

Navigation Menu

Repository navigation

code-index: graph publication opens a segment file per page read and never finishes under endpoint scanning #4532

Observed

Why it matters

Expected

53 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

code-index: graph publication opens a segment file per page read and never finishes under endpoint scanning #4532

Description

Observed

Why it matters

Expected

Activity

ScriptedAlchemy commented on Nov 8, 2026

ScriptedAlchemy commented on Nov 8, 2026

ScriptedAlchemy commented on Nov 8, 2026

ScriptedAlchemy commented on Nov 8, 2026

ScriptedAlchemy commented on Nov 8, 2026

53 remaining items

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions