Repository navigation
perf(code-index): reuse staged read and durability handles - #4792
ScriptedAlchemy wants to merge 97 commits into
Conversation
|
|
#4533 owner / sweep coordination (partial PR, |
|
#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure. The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader. Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used. The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements. Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes. |
Keep the concurrent formatting commit in the branch history. Rehome and extend its production-reader regression to cover in-place mutation, transaction poisoning, failed reads, and escaped-reader cleanup. Refs #4533
Keep the first-open file identity for the preparation even after its bounded read cache evicts the descriptor. Refuse a replaced pathname before another open, including across the cache's eviction boundary. Prove the refusal with a real nine-segment regression. Refs #4533
Exercise a real staged generation's whole and repeated range reads, verify the content digest and exact bytes, then publish and cold-decode its durable snapshot without changing the production storage format. Refs #4533
|
#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits |
Quote the pathname API in documentation and retain exact empty-buffer regressions using equality assertions required by the pinned Clippy policy. Refs #4533
|
#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head |
|
#4533 owner — production fix published at 3286852 on this PR. Appended with Changed Added seven real-filesystem regressions in Verified generated targets: Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used. This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work. |
Reuse duplicated writer descriptors for Linux durability windows instead of reopening every segment. Preserve concurrent member fsyncs and refuse missing, replaced, or mutated members, including earlier flushed windows. Add kernel-observed open-count, barrier, ownership, and failure regressions. Refs #4533
|
Published the direct GitHub source repair at 4d0ad18, appended with The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not. Added Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval. The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533. |
Validate ranges and named extents before changing staged-reader LRU state. Reuse admitted metadata across the open, retaining replacement checks. Add a real full-cache regression for cold refusals and eviction order. Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the same-inode durability regression without changing its assertions. Refs #4533
|
#4533 owner — hosted gate repaired on the existing PR (no merge). At |
Publish missing canonical names with atomic no-replace semantics. If another publisher wins, verify byte identity before deduplicating and preserve any conflicting immutable content. Exercise the production path for conflicting, identical and absent destinations. Refs #4533
On Unix use the directory device/inode identity directly when preparing and validating a publication. Preserve existing Windows handle identity checks and reject substituted directories before commit. Add a real prepared publication regression for changed and restored directory identities. Refs #4533
|
Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:
Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work. Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it. |
|
#4533 owner-lane / sweep and PR-maintenance coordination — published, partial, HOLD MERGE. Preserved concurrent work on the existing master-targeting #4792 and published two guarded nonforce commits: 9ea8a39, exactly the five hosted Bazel formatting hunks without weakening assertions, and 61394f7, wiring retained-history descriptor release into normal exclusive-lock generation retention. The new real two-generation Linux regression requires a positive cached descriptor, then zero process file descriptors to the unlinked inode, stale-read refusal and active-snapshot correctness. Hosted CI 42712609686 is queued; no current-head passing Rust tests or independent approval. The retained-inode review finding has a response but is not marked resolved. Please coordinate touching the seven updated files with this owner lane, keep PR unmerged and issue open. Packed O(1) cold first-opens, full daemon audit, managed endpoint before/after measurements, and independent approval remain outstanding. No Cargo, forced history, duplicate PR, external delegation, Kev/draft #4776 or release #4763 modifications. |
A historical page read still opened the active pointer for every range, even when its segment was cached. Pin the authenticated Unix pointer inode in the existing digest memo, rehash its bytes through that descriptor and verify path/fd identity before and after reads. Atomic publication changes fall back to cold verification; same-inode mutations refuse corruption. A Linux native inotify production-history test requires one cold pointer open and zero more across 17 exact-byte reads and a cloned store, with a positive kernel-open control. Refs #4533; partial, do not merge.
|
#4533 owner-lane checkpoint — published source and Bazel CI repair (Refs #4533; partial; HOLD MERGE). Two verified non-force, expected-head fast-forward commits on the existing
The existing real-production-publication Linux inotify regression is strengthened: one kernel open for cold pointer authentication, zero additional pointer opens for 17 historical whole/range reads and a cloned store, plus a positive independently observed pointer Additional audited blocker: Independent approval, current-head nonzero Bazel checks, full daemon paths (restore/reseat, verification, retention, transcripts, SQLite temp, response handles), snapshot/retirement proofs, and same managed-host syscall-open/latency before-after are pending. Coordinate further edits to the existing issue branch; do not merge or close #4533. No Cargo, force/rewrite, test relaxation, delegated engineering, decision-model/Kev/#4776 or release/#4763 changes. |
Clear the shared pointer memo before the destructive reset, retaining its mutex through deletion. Later reset failures must not retain an unlinked pointer descriptor across store clones. Add a real generation publication, contended-reset preservation, post-unlink failure and fresh-store recovery regression. Linux observes actual held inode descriptors before and after reset. Refs #4533
|
Review repair published: 0accee86, appended to fba00d6 via exact expected-head, nonforce update. Two files only: pointer-memo reset ordering in publication_store.rs and the new owning-runtime publication_reset.rs regression. This addresses the independently reported deleted-pointer descriptor leak on reset failure. The memo releases its descriptor before unlink and stays locked through destructive reset; the test covers lock-contention preservation, a real post-unlink failure, positive/zero Linux inode-descriptor observations across clones, and fresh-store exact snapshot recovery. Hosted Bazel CI #42723182058 is queued for this head. No current-head passing Rust/format/Clippy result or independent approval is claimed. The review thread has been answered and is left for independent re-review. Preserve this head unless there is an actionable source or CI correction; do not churn pending validation. Refs #4533 only. The distinct-segment packed-storage cutover, per-page generation-lock opens, complete daemon open-count audit, and managed-endpoint measurements remain outstanding. Keep this PR unmerged and #4533 open. No Cargo, force/rewrite, test weakening, infrastructure, protected Kev/decision-model or release changes. |
Exercise the production metadata reader for vanished/directory path errors, retain sibling-test visibility of the bounded descriptor cache, and supply the absent optional retirement callback in the isolated text-artifact test. Apply the exact hosted Rustfmt hunk to the failed-reset regression. These repair compilation and formatting without removing assertions or changing storage behavior. Refs #4533
The retained cache's sibling unit test reads its bounded length. Match the new scheduler-private field visibility on its element type to avoid a private-interfaces warning under Bazel Clippy -Dwarnings. Refs #4533
|
#4533 owner/sweep coordination — published compiler/format repairs; Refs #4533 only. New head ec15c307 after 63a01e7, both with exact remote parents and nonforce expected-head ref updates. These repair three confirmed compiler-failure families from Bazel #42716161880 and one exact Rustfmt hunk from #42723182058, with no test deletion or weakened assertions. Current-head Bazel CI #42726694390 queued, NOT green. Please preserve this head for verification; coordinate overlapping maintenance edits. Packed/O(1) distinct cold opens and per-page generation lock overhead, daemon-wide open-count audit, managed endpoint measurements and independent approval remain outstanding, so do NOT merge #4792 or close #4533. No Cargo, protected Kev/#4776, release/#4763 or infrastructure action. |
Retain the missing-generation and non-file refusal assertions while avoiding expect_err's Debug bound on the metadata success type. Remove a Clippy redundant closure in the live retained-segment descriptor regression. Refs #4533
Keep a per-store, clone-shared Unix lock descriptor while taking and dropping an actual kernel shared lease for each historical page. Authenticate the lock pathname inode before and after acquisition; fail closed on replacement and retry rather than block a contending in-process reader. The Linux production read regression now counts actual lock-file opens across 17 reads and a clone, including an independent positive event-observer control. Refs #4533
Exercise real Unix kernel shared and exclusive leases across cached reads, including same-process contention, explicit release, recovery and a replaced lock pathname. A stale cached descriptor must refuse the new inode. Refs #4533
|
#4533 owner/sweep coordination — published partial fixes (HOLD MERGE). Branch head 30747aaa, exact-parent expected-head nonforce chain from ec15c30. 46c01415 repairs confirmed hosted Bazel Clippy/test compilation without weakening error assertions. a840b5c9 removes per-historical-page generation-lock Exact-head hosted Bazel CI #42737264260 queued, NOT passed; native macOS skipped. Related PRs #4796 (graph), #4789 (sparse evidence), and #4846 (graph proof) have no direct changed-path overlap with this lock/cache work. Do not merge or close: O(distinct cold segment) opens still requires packed/offset storage or equivalent, full daemon open-path audit, managed-endpoint measured latency and independent approval. No Cargo, force/rewrite, protections bypass, protected Kev/decision-model (#4776) or release (#4763) changes. Refs #4533. |
Authenticate the lock-file directory entry with lstat rather than following a symlink to a retained descriptor. A same-inode symlink replacement must not silently authorize historical reads under an old lock authority. Retain the one-open cache and kernel fencing. Extend the real Unix kernel regression with a positive same-inode symlink control, refusal, exact-inode restoration, and writer exclusion. Refs #4533
|
#4533 owner lane / sweep coordination — published narrow lock-authority correctness fix (Refs #4533; partial, HOLD MERGE). Published Confirmed source flaw: The new reused historical-read generation-lock descriptor compared Nonvacuous regression: Existing real Unix kernel-lock test now creates both a different-inode replacement and a same-inode symlink replacement; it proves the latter resolves to the exact cached dev/ino, requires typed refusal, restores the original inode, reacquires a real shared lock, and proves an exclusive writer is still excluded. The prior head fails the symlink-refusal assertion. Source and regression are committed; they have not been shown passing under Bazel on this SHA. Current-head hosted Bazel run #42744217678 is queued, native macOS not selected; independent approval and nonzero current-head test/Clippy evidence remain required. The architectural O(distinct segments) cold-open bound, full daemon-wide OS open audit, and same-managed-endpoint latency comparisons remain outstanding. Do not merge/auto-merge #4792 or close #4533. Preserve this head and coordinate any overlapping sweep edits. All Kev/decision-model work, including #4776, remains DRAFT; release #4763 read-only. No Cargo, test weakening, external delegation, force updates, releases or infrastructure changes. |
Reject both existing-target and dangling symlinks before pointer CAS or rename. The cold authenticated pointer recheck also uses lstat to forbid a symlink substitution to the same opened inode, without another open. Exercise real symlink pathname refusal, untouched target/link identity, and exact-byte ordinary publication in the owning runtime tests. Refs #4533
|
#4533 owner/sweep coordination: published narrow pointer-slot safety fix (Refs #4533, HOLD MERGE). 20c5a356 fast-forwarded the existing master-targeting PR from 53b765c with exact expected-head and no force. The commit refuses dangling/matching-byte symlink pointer slots during CAS and closes the cold-authentication final symlink-follow gap; it adds a real Unix successful/denied CAS regression. Hosted Bazel CI #42747934032 is queued; tests are not yet passing. PR #4796 remains the separate graph open path lane—no duplicate #4533 PR or overlap edits. Keep #4792 unmerged until independently reviewed, nonzero CI plus O(distinct segments)/managed-endpoint acceptance, and final-V2 snapshot/retirement proofs. Do not merge Kev/decision-model work (including #4776) or mutate release #4763. |
Fail closed when the final cold publication-pointer witness detects a replacement inode, symlink alias or changed bytes. Previously the reader skipped memo installation but still returned the stale parsed pointer. Exercise a real opened inode against same-byte replacement and symlink alias, followed by valid restoration, without an additional production pathname open. Refs #4533
A real same-inode rename updates Unix ctime. Refresh that metadata for the positive restoration control while preserving the same open descriptor; keep the byte-identical inode replacement and symlink refusal assertions. Refs #4533
Refuse a cold pointer after a pathname rotation without returning stale bytes. Treat legitimate concurrent replacement or unlink as a retryable CAS conflict, not a destructive corruption reset; retain corruption for symlinks and in-place byte or inode metadata mutation. Exercise missing path, same-byte new inode, symlink alias, and restoration through real descriptor metadata with no additional production open. Refs #4533
|
#4533 owner/sweep coordination (Refs #4533, partial): reviewed the current root and relevant nested AGENTS.md inventory, complete issue/comments, master reader, related #4796/#4547, review threads and CI. Published |
Apply the seven exact formatting hunks emitted by the current-head Rustfmt gate for retention and runtime, preserving all behavior and nonzero assertions. Refs #4533
|
#4533 owner/sweep coordination — published hosted formatting repair, Refs #4533 only (HOLD MERGE). Published 33df990 on existing #4792, exact child of e86a4e5 with a non-force Current-head Bazel CI #42758341694 is pending and not a passing test result; native macOS is skipped by this workflow. The open issue remains O(distinct segments) cold first-opens, final-V2 packed/offset-or-equivalent storage, daemon-wide per-operation opens, managed-endpoint before/after latency, and independent approval. Preserve this head in other sweep tasks; no rebase/duplicate/merge/auto-merge/issue closure. Kev/decision-model work including #4776 stays DRAFT; release #4763 read-only. No Cargo, external delegation, or infrastructure actions. |
Recheck every pending member after all fsync workers join, before a window can succeed or release its retained descriptors. Per-member post-barrier checks alone miss changes while another worker finishes. Cover same-byte inode replacement during final sync and same-inode mutation during rollover with real fsyncs and deterministic, test-only completion handshakes. Preserve failed admission and exact recovery. Refs #4533
|
Pushed b852ca0 ( Found a separate pending-window durability race: an early member could change after its own fsync and identity checks while the concurrent window was still running. The fix checks every pending descriptor and pathname after all workers join, before success or rollover promotion. It adds no pathname reopens or fsync calls and preserves the existing bounded concurrency window and failed-admission ownership. Added two owning-crate regressions in Validation: reviewed the final commit diff and existing Bazel unit-test wiring. No local Bazel/Rust test execution was available. Exact-head hosted CI run 42763318601 is queued: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42763318601. The regressions are committed, not yet established passing. No merge or issue closure; this does not claim full #4533 acceptance. |
Apply the three exact rustfmt hunks reported by current-head Bazel repository gates for the durability window regressions. Preserve all production logic, test assertions and deterministic interleavings. Refs #4533
|
Refs #4533 — current-head owner checkpoint (PARTIAL / HOLD MERGE). Published Read root AGENTS.md/current issue (no comments), head and master storage/retention/test implementation, related #4532 PR #4796, all visible #4792 review submissions and threads, and hosted CI. The existing #4792 is the appropriate issue PR; no competing #4533 PR found. One cancellation-order review thread remains independently unresolved (source correction present); no approving independent review is recorded. Merge hold regardless of CI: #4533's fundamental 18k+/73k+ distinct-segment cold-open problem remains O(segments); the existing bounded reader cache and reuse of lock descriptors do not prove O(1) cold first-opens. Packed/offset-indexed storage or another demonstrated bound, a complete daemon-wide per-operation open audit (including restore/reseat, retention, transcript, SQLite temp and response paths), byte-exact final-V2 snapshot/corruption/retirement proof, same-machine managed-endpoint before/after open counts and latency, and independent approval remain outstanding. Do not merge/auto-merge or close #4533. Kev/decision-model including draft #4776 remains protected; release #4763 remains read-only. No Cargo, Kody/Copilot/external delegation, infrastructure or release action. |
Refuse symlinked generation names and inode or metadata changes across full and metadata-only retention scans. Keep the same single pathname open while validating the original descriptor and directory entry. Add owning-crate real-file regressions for an alias and a same-byte foreign-inode rotation during both verification modes, plus a healthy full-hash control. Neither path requires additional open calls. Refs #4533
|
Refs #4533 — published code-index retention correctness repair on the existing master-targeting PR, preserving peer work. Exact fast-forward: Confirmed source defect: The retention metadata scanner accepted a symlink under a generation-shaped pathname: Fix: Falsifiable owning-crate regressions: A genuine content-addressed generation plus a symlink alias is rejected in both modes while the original still decodes with its exact full SHA-256; a same-byte, different-inode atomic replacement at the third real cancellation checkpoint (after the first descriptor read) must fail in both modes, retaining replacement bytes. Old code accepts these reads. These are published, not yet executed; no local Bazel checkout/DNS. New-head hosted CI #42777818917 is pending. Previous-head #42768671504 passed Repository gates and Dashboard but native/Clippy jobs were superseded by this push, not green. HOLD MERGE / PARTIAL: The primary O(distinct sealed segments) cold first-open problem is still unsolved: packed/offset storage or a demonstrated equivalent, full daemon open-path audit, managed endpoint-security before/after measurements, snapshot/retention proof and independent approval are outstanding. Please preserve this exact branch in general PR sweeps; do not duplicate, auto-merge, merge, rewrite or close #4533. Kev/decision-model (including #4776) stays draft, #4763 release read-only. No Cargo, external delegation or infrastructure changes. |
Nov 9, 2026 — #4533 cold-pointer authority review (PARTIAL / HOLD MERGE)
Published head: e86a4e5, extending the existing issue PR by three exact-parent, non-force expected-head updates: ebb9f1d, fcbe281, and e86a4e5. Refs #4533 only.
New confirmed source defect: A cold pointer read that detected an inode/byte mismatch during its final authenticated-descriptor readback only skipped memo installation and still returned the stale parsed pointer. The Unix cold path now refuses instead. A changed/missing pathname (including a legitimate concurrent atomic pointer rotation) yields a retryable
CompareAndSwap, avoiding destructive reset of a healthy new pointer; symlink/nonfile or same-inode byte/metadata mutation yields reset-required corruption. It reuses the already opened descriptor and existing stat witnesses; no additional production open().New owning-crate regression: A real nonempty original pointer, a same-byte different-inode replacement, a missing pathname, a symlink alias resolving to the original open inode, and exact inode restoration exercise the same production final-witness function. Positive original/restored cases succeed, rotation/missing cases must return CAS, and symlink must return reset-required corruption. The restoration control refreshes legitimate Unix ctime after rename, rather than comparing the pre-rename timestamp. Source is published, but not yet proven passing on hosted Bazel.
Validation HOLD: current-head Bazel CI #42752854848 queued; no current-head nonzero test, fmt or Clippy pass, independent approval, or native macOS evidence. Architectural O(distinct cold segments) still requires packing/offset addressing or an equivalent proven bound; daemon-wide open counts, final-V2 retention/read proof, and managed-endpoint before/after measurements remain outstanding. Do not merge, auto-merge or close #4533. Kev/decision-model PRs (including #4776) stay DRAFT and unmerged; release #4763 remains read-only.
November 9, 2026 — latest #4533 owner checkpoint (PARTIAL; HOLD MERGE)
Published exact head: 20c5a356, non-force fast-forward from 53b765c with an expected-head guard; existing PR to
master, no duplicate. Refs #4533 only.New correctness repair: The active-pointer commit CAS preflight used
metadata(), following symlinks. A symlink to a matching foreign pointer or a dangling target could therefore be treated as a valid/absent pointer slot, andtempfile.persistwould replace the symlink.require_regular_pointer_slotnow usessymlink_metadata()to reject both shapes as reset-required corruption. Cold pointer authentication's final pathname witness also usessymlink_metadata()rather than following a symlink to the same opened inode. No additional pathname opens, no bypass of the existing store lock or byte comparisons.Falsifiable nonzero regression: The runtime owning-crate test exercises both existing-target and dangling symlink slots against the real commit CAS, requires typed corruption and unchanged pathname/foreign bytes, then restores an ordinary pointer file and requires exact-byte publication. The old code admits the symlink and fails the refusal assertion. The test is committed but not yet executed on the new head.
Validation HOLD: hosted Bazel CI #42747934032 queued at inspection. Earlier runs were cancelled/superseded; no new-head passing Rust, format, Clippy, or native macOS claim. Independent approval absent. The central O(distinct cold segments) file-open bound, audit of all daemon open paths, final-V2 pack/offset/read/retention proof and managed-endpoint before/after measurements remain incomplete. Do not merge, auto-merge or close #4533. Kev/decision-model (including draft #4776) and release #4763 are unchanged.
Nov 9, 2026 17:28 PT — issue owner update (Refs #4533, PARTIAL / HOLD MERGE)
Published head: 30747aaa on the existing
master-targeting issue PR. Two parent-linked nonforce expected-head updates: 46c01415 and a840b5c9, then the kernel-exclusion regression on 30747aa. No rewritten or discarded peer work and no duplicate PR.Confirmed compiler fixes: Hosted Bazel run #42726694390 failed Clippy/test compilation:
expect_erronSealedGenerationManifestMetadataV1required an unimplementedDebugbound, and a redundant closure was denied by Clippy. 46c0141 preserves both error-kind assertions while using.err().expect(...), and usesResult::okin the real retired-segment fd test. Repository gates passed on the predecessor; no current-head passing claim.New measured read-path improvement: Historical range/whole requests had still reopened
.code-generation-retention.lockon every page. The Unix retained-read path now owns one clone-shared cached lock-file descriptor, but obtains and explicitly releases a new kernel shared lock per read, serializing same-process lease ownership, verifying pathname/inode identity before and after acquisition, and refusing replaced lock authorities. A Linux inotify production-read regression requires one cold lock-file open over 17 historical reads, zero additional opens from a cloned store, and a positive real-open observer control, while existing pointer/segment exact-byte checks remain. A separate real-kernel regression verifies concurrent exclusive-writer refusal, release/reacquire, and stale-lock-inode refusal. Not a global lock bypass or a claim that distinct-segment cold opens are solved.Validation HOLD: current-head hosted Bazel #42737264260 queued. Do not merge or close until nonzero Bazel tests/Clippy and independent review pass; native macOS is skipped by the workflow. Core #4533 still requires packed/offset storage or equivalent distinct-cold-segment bound, daemon-wide open path counts, on-device managed-endpoint before/after latency, and final-V2 snapshot/retirement proof. Protected Kev/decision-model work (#4776 included) remains draft; #4763 read-only; no Cargo, Kody/Copilot, release, secrets or infrastructure actions.
Current owner checkpoint — Nov 9, 2026 (Refs #4533, partial; HOLD MERGE)
Published head: ec15c307, following 63a01e7, on the same master-targeting issue branch. Both are exact-parent non-force expected-head fast-forwards, not rewrites. No duplicate issue PR.
Confirmed CI source blockers corrected: hosted Bazel #42716161880 failed native Linux/Clippy compilation for two stale revision-reader test calls, two private sibling-test cache field accesses, and a missing eighth optional retirement argument in an isolated maintenance test. Hosted #42723182058 found one exact Rustfmt hunk in the new reset regression (all eight repository policy tests passed). 63a01e7 updates tests to the production metadata-open reader without deleting assertions; aligns the cache field visibility, supplies None for the test-only absent callback, and applies the exact rustfmt diff. ec15c30 aligns the cache element type's scheduler-private visibility to prevent a private_interfaces warning. No production behavior change, weakened test, Cargo, CI workflow or infrastructure mutation.
Validation is NOT GREEN: Current-head hosted Bazel CI #42726694390 queued at inspection. Rust tests, Clippy, format and independent approval must pass on this head, not old runs. Native macOS remains skipped. Central acceptance is still open: O(distinct cold segments) first-opens, generation lock opens per historical page, full daemon path audit, correct packed/offset-index representation or equivalent, and same managed-endpoint before/after open counts and latency. Do not merge/auto-merge or close #4533; protected Kev/decision-model #4776 stays DRAFT and release #4763 read-only.
Current ownership checkpoint — November 9, 2026 — Refs #4533 (partial; HOLD MERGE)
Published head:
61394f7, on existing issue PR targeting master. Exact remote parent9ea8a39, non-force update guarded byexpected_sha. A concurrent peer's separate retention commits were preserved; no duplicate PR, rewrite, release or protected model work.New source repair: Normal code-generation retention unlinked historical segments while the daemon publication-store's newly added eight-entry cross-request reader cache could still hold the deleted inodes. Unix therefore retained actual allocated segment bytes indefinitely on an idle daemon. The production mounted/retiring registry now provides its shared reader-cache release to the canonical maintenance executor. Under the exclusive generation/segments locks and after verifying the active pointer, the executor releases authenticated read handles before any segment unlink; a poisoned cache refuses deletion. Read-path locking, pointer/index authority and dry-run behavior remain unchanged. Applies to both mounted and registered maintenance paths. Every direct executor test caller was migrated.
New falsifiable regression: A real two-generation source publication with an old-only nonempty segment opens that historical segment through the production cached read, proves its actual Linux process descriptor pins the inode, executes normal authenticated retention with the callback, then requires one callback, closed descriptors, absent old-only pathname, no remaining process FD to its inode, refusal to serve its retired identity, and the active generation still readable. This test is committed, not yet Bazel-verified.
Hosted formatting repair: preceding CI #42706670785 passed all eight script checks but failed the two Rustfmt targets in exactly five test-source hunks. Commit
9ea8a39applies only those five emitted hunks; no production behavior or assertions were changed. The old CI's Linux/Clippy jobs were superseded, not passed.Validation HOLD: current-head hosted Bazel CI #42712609686 queued; no current-head native Rust, Linux fleet, formatter, Clippy or macOS passing claim. Independent approval absent; ordinary workflow skips macOS. The central O(distinct cold segments) issue is still not solved: packed/offset-indexed generation storage (or formally proven equivalent bound), full daemon open-path audit, snapshot/retirement/durability proof on the final representation, and same managed-endpoint before/after syscall and latency measurements remain required. Do not merge #4792 or close #4533. Keep Kev/decision-model work (including mixed draft #4776) draft and release #4763 read-only.
Current owner update — 2026-11-09 (Refs #4533; partial, DO NOT MERGE)
Published head:
b91f221on the existingmaster-targeting PR. Exact expected-headdacc5fc62f626af2334df7d40cfa206e0ab883dd, non-force fast-forward, no duplicate PR.Confirmed native Bazel evidence on predecessor
dacc5fc: hosted run #42677767864 executed and passed//crates/tracedecay-code-index-runtime:unit_test(68.9s), including the new Linux native inotify open-count/cold-snapshot/inode-substitution regression target, and//crates/tracedecay-private-fs:unit_test(0.8s); Clippy, Dashboard, change-scope and benchmark checks passed. Linux fleet was NOT green:tracedecay:dashboard_api_testandtracedecay-global-db:unit_testeach timed out at 300s. All eight repository policy tests passed, but the Rustfmt genrule failed on exactly two formatted assertions instaged_segment_reads.rs; Feature gates inherited this red.b91f221applies only those two exact hosted rustfmt hunks without changing behavior or assertions. New-head CI #42695862696 is pending; no current-head green claimed. Native macOS has been skipped.Acceptance HOLD: A bounded eight-descriptor warm cache avoids repeated page opens within one reader but still cold-opens O(distinct sealed segments). Implement packed/offset-indexed generation storage or demonstrably equivalent O(1) first opens, fully audit daemon opens (verify, restore/reseat, retention, transcripts, SQLite temporary storage, response handles), prove integrity, snapshots and retirement, and compare syscall open counts/latency on the same managed endpoint-scanned machine. Independent approval remains outstanding. No
Fixes #4533, merge/auto-merge, issue closure, Cargo, external delegation, forced update, infrastructure or protected Kev/#4776 or release/#4763 mutation.Previous owner update — 2026-11-09 (Refs #4533, partial)
Published head:
dacc5fc, appended to the existingmaster-targeting #4792 branch using an exact expected-head non-force fast-forward; no duplicate PR.New source improvement: The retained-generation
segment_readerpreviously opened a pathname for every whole/range read, even for multiple pages in the same segment. On Unix, each reader now keeps an 8-descriptor bounded LRU for its lifetime; a cache hit verifies both the path and opened inode, and post-read validation clears any corrupted/changed handles and output. Cold misses admit a validated new handle before evicting another. Non-Unix behavior is unchanged. This is a real open reduction for repeated pages, not a packed-store or O(1) distinct-segment cold-open claim.New regression: A Linux production publication + cold snapshot test uses an inotify-backed native observer to demand exactly 1 kernel pathname open for 1 whole + 16 range reads, validates all returned bytes and cold snapshot identity, controls the observer with an additional observed open, and rejects a same-byte/different-inode pathname replacement without exposing stale data or reopening it. Previous code opens that file 17 times and does not retain the inode witness. The regression is published but is not yet confirmed passing on hosted Bazel. Current-head hosted CI was queued on inspection; native macOS is skipped by the workflow.
Merge/closure hold: Independent approval, packed storage (or equivalent proof of O(1) cold first opens across distinct segments), complete daemon-path open-count audit, snapshot/retirement proof, and same-managed-endpoint before/after measurements remain outstanding. Do not merge/auto-merge #4792 or close #4533 yet. Protected Kev/decision-model including draft #4776 and release #4763 are untouched. No Cargo, Kody/Copilot, forced history, gate bypass, or infrastructure action.
Current head — owning-crate durability regression + hosted format repair (2026-11-09)
Published, Refs #4533 (partial): 33d1b0c applies exactly the two Rustfmt hunks from hosted Bazel repository gates #42649192033; it changes no assertions or production behavior. 0384da6 responds to the new independent review finding that the final-barrier regression had left its owning crate. It moves the regression into
tracedecay-private-fs's Bazelunit_test, deleting the duplicate runtime-only source. The existing test-helper fsync-latency fixture now permits a channel-controlled pause at the real sync boundary, avoiding a fragile sleep, cyclic runtime-core dependency, mocked fsync, or tracing subscriber policy bypass. The new nonzero test requires 33 real nonempty writers, a real observed fsync, same-byte/different-inode replacement after preflight, typed refusal, retained ownership, exact bytes and refusal without further fsync on retry.Exact branch head:
0384da63eacf27f7973fc891ce3dccc6ea0fcba0, basemaster, expected-head non-force fast-forward. New hosted Bazel run #42655847587 was queued on inspection; no current-head Rust, formatting or Clippy green is claimed. Native macOS skipped. The review thread has been answered but remains pending independent re-review and positive owning-crate test execution. Prior run #42649192033 failed only the two reported Rustfmt lines in the code-index durability test at its Repository gates (8 policy targets passed); do not describe it as green.No merge/auto-merge/closure: O(distinct-segment) first-opens persist, requiring real packed/offset-index storage or equivalent, all daemon-path open counts, byte-exact snapshot and retirement proofs, and measured managed endpoint-scanned before/after latency. No protected Kev, #4776, release #4763, or infrastructure changes.
Latest hosted Bazel repairs (2026-11-09) — Refs #4533
Current head:
3d12658, existing issue branch targeting master, exact-head non-force fast-forward from8e17075. New hosted Bazel run #42649192033 is queued; no current-head CI green is claimed.At the previous exact head, run #42631275844 confirmed a rustfmt failure in the latest immutable-segment source/tests and Clippy failures for four disallowed lone
tracing::subscriber::with_defaultinstalls and twoassert_is_emptyusages. It also executed 182 of 183 Linux fleet Bazel targets successfully, including noncached//crates/tracedecay-code-index-runtime:unit_test(69.5 seconds) and//crates/tracedecay-private-fs:unit_test(1.0 second); the unrelated//crates/tracedecay-global-db:unit_testtimed out after 300 seconds. Those passes belong to the prior head, not the new revision. Dashboard, benchmark self-tests, and change-scope passed; feature gate was blocked by rustfmt; native macOS was skipped. The timeout's root cause remains unproven.Commit
52b335dapplies the exact hosted rustfmt changes, preserves the existing source protections, and uses the canonicaltracedecay_runtime_core::logging::set_tracing_capturein the staged-open and immutable-collision regressions. The Linux final-barrier test was moved from the filesystem kernel crate to its actual code-index runtime caller so it can use that sanctioned capture helper without introducing a cyclic kernel dependency, unsafe scoped trace interest, suppressed Clippy policy, or a test-only production hook. It still executes real fsync on a nonempty 33-file durability batch, substitutes a distinct inode during the final barrier, and requires anInvalidDatarefusal, exact bytes, and refused no-fsync retry. The existing private-fs batch tests continue to assert 32+1 real fsync-barrier counts and ownership. Commit3d12658formats the moved test's retry assertions. No Rust dependencies, generated rules, CI workflows, or unrelated infrastructure changed. The new moved test and whole head still require nonzero Bazel verification.Merge/closure hold: The core acceptance remains incomplete: distinct sealed segments still cold-open O(segments) pathnames. Implement packed/offset-based storage or prove an equivalent bound, audit verification/restore/reseat/retention/transcripts/SQLite/response paths, prove byte-exact snapshot/read/corruption/retirement/durability, and measure before/after OS-level opens and wall time on the same managed endpoint-scanned large repository. Independent approval and native macOS are also absent. Do not merge/auto-merge #4792 or close #4533 based on partial improvements; no Kev/decision-model or release changes.
Published: staged inode authority across canonical deduplication (2026-11-09)
Commit
8e17075closes a source correctness race: a private segment inode is authenticated during staging, but the canonical-collision comparator independently opens and measures its current pathname. If the staging pathname is replaced with another same-byte/different-inode file after the earlier witness check but before comparator metadata acquisition, byte equality alone previously authorized deletion of the original staged copy. Both the existing-canonical path and atomic no-replace collision path now recheck the originally authenticated temporary inode after equality, before discarding the staging copy. The checks are metadata-only, adding zero pathname opens. The new outer immutable-path comparison tracing span measures the endpoint-authorized opens; the original inner byte-read span is preserved.A production-writer/durability-path regression stages nonempty SHA-256-addressed bytes, installs a competing identical canonical name, then uses the new production comparison span to deterministically swap the temporary inode after the initial witness check and before the comparator samples metadata. It requires a nonzero real comparison, typed refusal, intact canonical bytes and both staging file contents, retained pending ownership, and retry refusal even after the old inode is renamed back (ctime changed). The predecessor accepts and deletes the replaced pending name. This test is committed, not yet verified passing. Exact-head hosted Bazel CI is queued; macOS native is not enabled. No Cargo, no force update, no CI workflow changes.
Refs #4533, partial only. Cold first-opens remain O(distinct segments); packed/offset storage, full daemon open-path audit, managed endpoint-scanned before/after measurements, independent approval, and on-master acceptance remain outstanding. No merge/auto-merge or issue closure.
Latest source fix — canonical deduplication identity (2026-11-09)
Commit
e18d295makes canonical/pending immutable-segment equality fail closed when either pathname or opened inode changes while a potentially slow comparison runs. Previouslyfiles_equalcould report equal opened bytes after a concurrent same-byte different-inode pathname swap, then authorize removal of the private staged segment. Unix checks the original pathname/fd identity before and after comparison, without extra pathname opens. A real nonempty same-byte replacement test swaps the canonical inode inside the production comparison span, demands refusal, checks exact bytes/inode and recovery. The predecessor would accept the comparison. Refs #4533; do not merge yet: the new regression and formatting are not validated on the current head, and O(1) cold first-opens and full endpoint acceptance remain outstanding.Refs #4533
Latest source repair — failed cold open leaves healthy LRU intact (2026-11-09)
Commit
2e1b241fixes a verified source-level cache-admission gap: after a valid ninth staged segment passed pathname metadata checks, the eight-entry pinned LRU evicted its oldest healthy handle beforeFile::open. If that pathname disappeared or open was refused in the intervening syscall, the healthy descriptor was lost, and a subsequent read paid another endpoint-authorized open. The candidate now opens and authenticates before the old descriptor is evicted. The retained cache stays at eight entries with one transient candidate at admission. Existing inode, digest, corruption, output-clearing, and poison checks are unchanged.The new Linux real-filesystem regression
staged_failed_cold_open_preserves_eight_healthy_handlesstages nine different nonempty content-addressed segments, proves eight successful opens with exact bytes, and deterministically moves the ninth file at the existing production open trace event after metadata validation but before its real syscall. It requires one observed failed open attempt, cleared output, all eight healthy handles retained, and zero further opens for the original oldest segment. Restoring the same inode then permits exact ninth-segment bytes; the predecessor evicts one handle and fails this test. The assertion is nonvacuous and uses production reader/writer behavior, not a mocked opener.Validation hold: the patch is published but has not been proven passing by Bazel at this head. Current hosted Bazel CI #42620737107 was pending at inspection; native macOS was not requested by that workflow. No Cargo was used. Independent approval, full Linux/Windows/native verification, packed O(1) cold storage, daemon-wide open-count audit and same-managed-endpoint before/after timing remain missing. Refs #4533 only; do not merge this partial PR or close the issue based solely on green summary gates.
Latest published repair — staged checksum failure poisons preparation
Commit
76b7556closes a fail-closed gap: a corrupt auxiliary staged segment could be opened after mutation, fail its first whole-segment SHA-256 read, yet remain cached and leave the preparation writable when the inode metadata stayed stable during the read. The reader now invalidates the entire open preparation on a failed authenticated read, clears the returned bytes and pinned handles, and refuses repeated requests without another pathname open. Neither restoring the segment bytes nor committing a valid-looking manifest can reauthorize the transaction.The new real-filesystem regression
staged_segment_cold_digest_mismatch_poison_survives_restorationdeliberately corrupts a nonempty staged segment before the first pinned open, so the SHA-256 check rather than a changing-metadata check detects the error. It requires a positive file-open count, cleared output, zero retained handles after the first failed read, no retry opens, refusal after restoration, and failed write/commit with no installed active generation. This is a new published regression, not yet proven passing; the previous implementation retains a handle and therefore fails its zero-handles assertion.Hosted Bazel CI run #42581012282 is queued on this head. No local Bazel executable/checkout is available, and no Cargo was executed.
Scope and current state
This remains a partial code-index storage improvement. It reduces redundant pathname opens and fixes integrity/durability failures, but does not yet implement packed storage or O(1) cold first-opens across distinct segments. Keep the PR unmerged and #4533 open on this PR's summary checks alone.
The PR now targets master, not draft #4837. Integration commits
4ea334eand835bd99retain the existing branch history and adopt the exact CI configuration already merged on master, ending at02e3c8b28b8dfc73d6fcf9b4e36120256dbe01c0. The five preceding storage feature files were preserved byte-for-byte during integration. The resulting proposed diff has six storage source/test files and no CI, workflow, dependency, or infrastructure changes. No action was taken on #4837 itself.Current head:
76b7556730ce493ed871f172a2c9639f04640163.Latest source repair: final durability barriers
Commit
e9b6e8efd8d7355b10a00739e7dfdd9aa7f45184fixes a gap in LinuxDurableFileBatch::sync: earlier flushed windows were checked only before the final pending window's potentially blocking fsyncs. An earlier pathname could be replaced during that last barrier and the batch could still report success.The final sync now checks earlier identities again after the pending barriers complete. The initial fail-fast checks, error propagation, descriptor bound and failed ownership remain intact. This adds metadata checks, not pathname opens or additional fsyncs.
The new real-filesystem regression
framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier:InvalidData, exact replacement bytes, retained batch ownership, and refusal on retry before another barrier.The preceding implementation returns success in this interleaving; the new assertion would fail it. This is a published regression, not yet an executed passing result. Existing batch and staged-reader tests are unchanged.
Existing production improvements retained
These are source-level improvements. They are not managed-endpoint wall-time measurements and do not eliminate the first open of each distinct segment.
Validation
Prior-head hosted Bazel run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42579869168 (superseded by the published checksum repair). New-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42581012282 . No current-head passing Rust, formatting or Clippy result is claimed here.
The prior head
b42f7ca396had successful hosted repository gates, benchmark-harness checks and Dashboard at the last inspection. Its Linux fleet was queued and Clippy was still running. An earlier owner report records 18 staged-reader tests and 33 private-filesystem unit tests passing under local Bazel on macOS at that prior head; that report is not evidence for this new Linux-only regression or the current head.Relevant generated Bazel targets remain unchanged:
Require a nonzero executed count and inspect the tested SHA. This environment has no local Bazel executable/checkout or outbound checkout DNS; no Cargo or direct Rust compiler/formatter execution was used. The available GitHub connector has workflow inspection/reruns but no native workflow-dispatch action. Native hosted macOS and current-head independent approval remain outstanding. Existing resolved inline review threads are not approving reviews.
Remaining issue acceptance
The implementation still needs the architectural packed/offset-index cutover (or a demonstrated equivalent), including every production writer, decoder, graph/lexical reader, sparse successor and retention caller. A fresh generation still has O(distinct segments) cold first-opens. Also outstanding: the complete daemon-path open-count audit (verification, restore/reseat, retention, session transcripts, SQLite temporary files and response handles), full snapshot/read/corruption/retirement/durability proofs, and before/after OS-level opens plus wall time on the same managed endpoint-scanned large repository.
Graph-specific work remains separately owned by #4796. Keep #4533 open until all acceptance is proven on master. No release, tag, production, credentials, protected Kev/decision-model changes, or external delegation.
The owner’s master synchronization subsequently moved this PR back to master at
835bd99377. Its six-file diff is confined to the original five storage paths and the added final-barrier regression. The current master cache snapshot is the second merge parent and the PR is mergeable. On this head all 18 staged-segment read tests and all 33 macOS private filesystem tests passed under Bazel. The final-barrier interleaving uses the Linux-only deferred fsync windows and awaits the hosted Linux target.