Skip to content

perf(code-index): reuse staged read and durability handles - #4792

Open
ScriptedAlchemy wants to merge 97 commits into
masterfrom
fix/4533-staged-segment-read-opens
Open

ScriptedAlchemy wants to merge 97 commits into
masterfrom
fix/4533-staged-segment-read-opens

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Nov 9, 2026 — #4533 cold-pointer authority review (PARTIAL / HOLD MERGE)

Published head: e86a4e5, extending the existing issue PR by three exact-parent, non-force expected-head updates: ebb9f1d, fcbe281, and e86a4e5. Refs #4533 only.

New confirmed source defect: A cold pointer read that detected an inode/byte mismatch during its final authenticated-descriptor readback only skipped memo installation and still returned the stale parsed pointer. The Unix cold path now refuses instead. A changed/missing pathname (including a legitimate concurrent atomic pointer rotation) yields a retryable CompareAndSwap, avoiding destructive reset of a healthy new pointer; symlink/nonfile or same-inode byte/metadata mutation yields reset-required corruption. It reuses the already opened descriptor and existing stat witnesses; no additional production open().

New owning-crate regression: A real nonempty original pointer, a same-byte different-inode replacement, a missing pathname, a symlink alias resolving to the original open inode, and exact inode restoration exercise the same production final-witness function. Positive original/restored cases succeed, rotation/missing cases must return CAS, and symlink must return reset-required corruption. The restoration control refreshes legitimate Unix ctime after rename, rather than comparing the pre-rename timestamp. Source is published, but not yet proven passing on hosted Bazel.

Validation HOLD: current-head Bazel CI #42752854848 queued; no current-head nonzero test, fmt or Clippy pass, independent approval, or native macOS evidence. Architectural O(distinct cold segments) still requires packing/offset addressing or an equivalent proven bound; daemon-wide open counts, final-V2 retention/read proof, and managed-endpoint before/after measurements remain outstanding. Do not merge, auto-merge or close #4533. Kev/decision-model PRs (including #4776) stay DRAFT and unmerged; release #4763 remains read-only.


November 9, 2026 — latest #4533 owner checkpoint (PARTIAL; HOLD MERGE)

Published exact head: 20c5a356, non-force fast-forward from 53b765c with an expected-head guard; existing PR to master, no duplicate. Refs #4533 only.

New correctness repair: The active-pointer commit CAS preflight used metadata(), following symlinks. A symlink to a matching foreign pointer or a dangling target could therefore be treated as a valid/absent pointer slot, and tempfile.persist would replace the symlink. require_regular_pointer_slot now uses symlink_metadata() to reject both shapes as reset-required corruption. Cold pointer authentication's final pathname witness also uses symlink_metadata() rather than following a symlink to the same opened inode. No additional pathname opens, no bypass of the existing store lock or byte comparisons.

Falsifiable nonzero regression: The runtime owning-crate test exercises both existing-target and dangling symlink slots against the real commit CAS, requires typed corruption and unchanged pathname/foreign bytes, then restores an ordinary pointer file and requires exact-byte publication. The old code admits the symlink and fails the refusal assertion. The test is committed but not yet executed on the new head.

Validation HOLD: hosted Bazel CI #42747934032 queued at inspection. Earlier runs were cancelled/superseded; no new-head passing Rust, format, Clippy, or native macOS claim. Independent approval absent. The central O(distinct cold segments) file-open bound, audit of all daemon open paths, final-V2 pack/offset/read/retention proof and managed-endpoint before/after measurements remain incomplete. Do not merge, auto-merge or close #4533. Kev/decision-model (including draft #4776) and release #4763 are unchanged.


Nov 9, 2026 17:28 PT — issue owner update (Refs #4533, PARTIAL / HOLD MERGE)

Published head: 30747aaa on the existing master-targeting issue PR. Two parent-linked nonforce expected-head updates: 46c01415 and a840b5c9, then the kernel-exclusion regression on 30747aa. No rewritten or discarded peer work and no duplicate PR.

Confirmed compiler fixes: Hosted Bazel run #42726694390 failed Clippy/test compilation: expect_err on SealedGenerationManifestMetadataV1 required an unimplemented Debug bound, and a redundant closure was denied by Clippy. 46c0141 preserves both error-kind assertions while using .err().expect(...), and uses Result::ok in the real retired-segment fd test. Repository gates passed on the predecessor; no current-head passing claim.

New measured read-path improvement: Historical range/whole requests had still reopened .code-generation-retention.lock on every page. The Unix retained-read path now owns one clone-shared cached lock-file descriptor, but obtains and explicitly releases a new kernel shared lock per read, serializing same-process lease ownership, verifying pathname/inode identity before and after acquisition, and refusing replaced lock authorities. A Linux inotify production-read regression requires one cold lock-file open over 17 historical reads, zero additional opens from a cloned store, and a positive real-open observer control, while existing pointer/segment exact-byte checks remain. A separate real-kernel regression verifies concurrent exclusive-writer refusal, release/reacquire, and stale-lock-inode refusal. Not a global lock bypass or a claim that distinct-segment cold opens are solved.

Validation HOLD: current-head hosted Bazel #42737264260 queued. Do not merge or close until nonzero Bazel tests/Clippy and independent review pass; native macOS is skipped by the workflow. Core #4533 still requires packed/offset storage or equivalent distinct-cold-segment bound, daemon-wide open path counts, on-device managed-endpoint before/after latency, and final-V2 snapshot/retirement proof. Protected Kev/decision-model work (#4776 included) remains draft; #4763 read-only; no Cargo, Kody/Copilot, release, secrets or infrastructure actions.


Current owner checkpoint — Nov 9, 2026 (Refs #4533, partial; HOLD MERGE)

Published head: ec15c307, following 63a01e7, on the same master-targeting issue branch. Both are exact-parent non-force expected-head fast-forwards, not rewrites. No duplicate issue PR.

Confirmed CI source blockers corrected: hosted Bazel #42716161880 failed native Linux/Clippy compilation for two stale revision-reader test calls, two private sibling-test cache field accesses, and a missing eighth optional retirement argument in an isolated maintenance test. Hosted #42723182058 found one exact Rustfmt hunk in the new reset regression (all eight repository policy tests passed). 63a01e7 updates tests to the production metadata-open reader without deleting assertions; aligns the cache field visibility, supplies None for the test-only absent callback, and applies the exact rustfmt diff. ec15c30 aligns the cache element type's scheduler-private visibility to prevent a private_interfaces warning. No production behavior change, weakened test, Cargo, CI workflow or infrastructure mutation.

Validation is NOT GREEN: Current-head hosted Bazel CI #42726694390 queued at inspection. Rust tests, Clippy, format and independent approval must pass on this head, not old runs. Native macOS remains skipped. Central acceptance is still open: O(distinct cold segments) first-opens, generation lock opens per historical page, full daemon path audit, correct packed/offset-index representation or equivalent, and same managed-endpoint before/after open counts and latency. Do not merge/auto-merge or close #4533; protected Kev/decision-model #4776 stays DRAFT and release #4763 read-only.


Current ownership checkpoint — November 9, 2026 — Refs #4533 (partial; HOLD MERGE)

Published head: 61394f7, on existing issue PR targeting master. Exact remote parent 9ea8a39, non-force update guarded by expected_sha. A concurrent peer's separate retention commits were preserved; no duplicate PR, rewrite, release or protected model work.

New source repair: Normal code-generation retention unlinked historical segments while the daemon publication-store's newly added eight-entry cross-request reader cache could still hold the deleted inodes. Unix therefore retained actual allocated segment bytes indefinitely on an idle daemon. The production mounted/retiring registry now provides its shared reader-cache release to the canonical maintenance executor. Under the exclusive generation/segments locks and after verifying the active pointer, the executor releases authenticated read handles before any segment unlink; a poisoned cache refuses deletion. Read-path locking, pointer/index authority and dry-run behavior remain unchanged. Applies to both mounted and registered maintenance paths. Every direct executor test caller was migrated.

New falsifiable regression: A real two-generation source publication with an old-only nonempty segment opens that historical segment through the production cached read, proves its actual Linux process descriptor pins the inode, executes normal authenticated retention with the callback, then requires one callback, closed descriptors, absent old-only pathname, no remaining process FD to its inode, refusal to serve its retired identity, and the active generation still readable. This test is committed, not yet Bazel-verified.

Hosted formatting repair: preceding CI #42706670785 passed all eight script checks but failed the two Rustfmt targets in exactly five test-source hunks. Commit 9ea8a39 applies only those five emitted hunks; no production behavior or assertions were changed. The old CI's Linux/Clippy jobs were superseded, not passed.

Validation HOLD: current-head hosted Bazel CI #42712609686 queued; no current-head native Rust, Linux fleet, formatter, Clippy or macOS passing claim. Independent approval absent; ordinary workflow skips macOS. The central O(distinct cold segments) issue is still not solved: packed/offset-indexed generation storage (or formally proven equivalent bound), full daemon open-path audit, snapshot/retirement/durability proof on the final representation, and same managed-endpoint before/after syscall and latency measurements remain required. Do not merge #4792 or close #4533. Keep Kev/decision-model work (including mixed draft #4776) draft and release #4763 read-only.


Current owner update — 2026-11-09 (Refs #4533; partial, DO NOT MERGE)

Published head: b91f221 on the existing master-targeting PR. Exact expected-head dacc5fc62f626af2334df7d40cfa206e0ab883dd, non-force fast-forward, no duplicate PR.

Confirmed native Bazel evidence on predecessor dacc5fc: hosted run #42677767864 executed and passed //crates/tracedecay-code-index-runtime:unit_test (68.9s), including the new Linux native inotify open-count/cold-snapshot/inode-substitution regression target, and //crates/tracedecay-private-fs:unit_test (0.8s); Clippy, Dashboard, change-scope and benchmark checks passed. Linux fleet was NOT green: tracedecay:dashboard_api_test and tracedecay-global-db:unit_test each timed out at 300s. All eight repository policy tests passed, but the Rustfmt genrule failed on exactly two formatted assertions in staged_segment_reads.rs; Feature gates inherited this red. b91f221 applies only those two exact hosted rustfmt hunks without changing behavior or assertions. New-head CI #42695862696 is pending; no current-head green claimed. Native macOS has been skipped.

Acceptance HOLD: A bounded eight-descriptor warm cache avoids repeated page opens within one reader but still cold-opens O(distinct sealed segments). Implement packed/offset-indexed generation storage or demonstrably equivalent O(1) first opens, fully audit daemon opens (verify, restore/reseat, retention, transcripts, SQLite temporary storage, response handles), prove integrity, snapshots and retirement, and compare syscall open counts/latency on the same managed endpoint-scanned machine. Independent approval remains outstanding. No Fixes #4533, merge/auto-merge, issue closure, Cargo, external delegation, forced update, infrastructure or protected Kev/#4776 or release/#4763 mutation.


Previous owner update — 2026-11-09 (Refs #4533, partial)

Published head: dacc5fc, appended to the existing master-targeting #4792 branch using an exact expected-head non-force fast-forward; no duplicate PR.

New source improvement: The retained-generation segment_reader previously opened a pathname for every whole/range read, even for multiple pages in the same segment. On Unix, each reader now keeps an 8-descriptor bounded LRU for its lifetime; a cache hit verifies both the path and opened inode, and post-read validation clears any corrupted/changed handles and output. Cold misses admit a validated new handle before evicting another. Non-Unix behavior is unchanged. This is a real open reduction for repeated pages, not a packed-store or O(1) distinct-segment cold-open claim.

New regression: A Linux production publication + cold snapshot test uses an inotify-backed native observer to demand exactly 1 kernel pathname open for 1 whole + 16 range reads, validates all returned bytes and cold snapshot identity, controls the observer with an additional observed open, and rejects a same-byte/different-inode pathname replacement without exposing stale data or reopening it. Previous code opens that file 17 times and does not retain the inode witness. The regression is published but is not yet confirmed passing on hosted Bazel. Current-head hosted CI was queued on inspection; native macOS is skipped by the workflow.

Merge/closure hold: Independent approval, packed storage (or equivalent proof of O(1) cold first opens across distinct segments), complete daemon-path open-count audit, snapshot/retirement proof, and same-managed-endpoint before/after measurements remain outstanding. Do not merge/auto-merge #4792 or close #4533 yet. Protected Kev/decision-model including draft #4776 and release #4763 are untouched. No Cargo, Kody/Copilot, forced history, gate bypass, or infrastructure action.

Current head — owning-crate durability regression + hosted format repair (2026-11-09)

Published, Refs #4533 (partial): 33d1b0c applies exactly the two Rustfmt hunks from hosted Bazel repository gates #42649192033; it changes no assertions or production behavior. 0384da6 responds to the new independent review finding that the final-barrier regression had left its owning crate. It moves the regression into tracedecay-private-fs's Bazel unit_test, deleting the duplicate runtime-only source. The existing test-helper fsync-latency fixture now permits a channel-controlled pause at the real sync boundary, avoiding a fragile sleep, cyclic runtime-core dependency, mocked fsync, or tracing subscriber policy bypass. The new nonzero test requires 33 real nonempty writers, a real observed fsync, same-byte/different-inode replacement after preflight, typed refusal, retained ownership, exact bytes and refusal without further fsync on retry.

Exact branch head: 0384da63eacf27f7973fc891ce3dccc6ea0fcba0, base master, expected-head non-force fast-forward. New hosted Bazel run #42655847587 was queued on inspection; no current-head Rust, formatting or Clippy green is claimed. Native macOS skipped. The review thread has been answered but remains pending independent re-review and positive owning-crate test execution. Prior run #42649192033 failed only the two reported Rustfmt lines in the code-index durability test at its Repository gates (8 policy targets passed); do not describe it as green.

No merge/auto-merge/closure: O(distinct-segment) first-opens persist, requiring real packed/offset-index storage or equivalent, all daemon-path open counts, byte-exact snapshot and retirement proofs, and measured managed endpoint-scanned before/after latency. No protected Kev, #4776, release #4763, or infrastructure changes.


Latest hosted Bazel repairs (2026-11-09) — Refs #4533

Current head: 3d12658, existing issue branch targeting master, exact-head non-force fast-forward from 8e17075. New hosted Bazel run #42649192033 is queued; no current-head CI green is claimed.

At the previous exact head, run #42631275844 confirmed a rustfmt failure in the latest immutable-segment source/tests and Clippy failures for four disallowed lone tracing::subscriber::with_default installs and two assert_is_empty usages. It also executed 182 of 183 Linux fleet Bazel targets successfully, including noncached //crates/tracedecay-code-index-runtime:unit_test (69.5 seconds) and //crates/tracedecay-private-fs:unit_test (1.0 second); the unrelated //crates/tracedecay-global-db:unit_test timed out after 300 seconds. Those passes belong to the prior head, not the new revision. Dashboard, benchmark self-tests, and change-scope passed; feature gate was blocked by rustfmt; native macOS was skipped. The timeout's root cause remains unproven.

Commit 52b335d applies the exact hosted rustfmt changes, preserves the existing source protections, and uses the canonical tracedecay_runtime_core::logging::set_tracing_capture in the staged-open and immutable-collision regressions. The Linux final-barrier test was moved from the filesystem kernel crate to its actual code-index runtime caller so it can use that sanctioned capture helper without introducing a cyclic kernel dependency, unsafe scoped trace interest, suppressed Clippy policy, or a test-only production hook. It still executes real fsync on a nonempty 33-file durability batch, substitutes a distinct inode during the final barrier, and requires an InvalidData refusal, exact bytes, and refused no-fsync retry. The existing private-fs batch tests continue to assert 32+1 real fsync-barrier counts and ownership. Commit 3d12658 formats the moved test's retry assertions. No Rust dependencies, generated rules, CI workflows, or unrelated infrastructure changed. The new moved test and whole head still require nonzero Bazel verification.

Merge/closure hold: The core acceptance remains incomplete: distinct sealed segments still cold-open O(segments) pathnames. Implement packed/offset-based storage or prove an equivalent bound, audit verification/restore/reseat/retention/transcripts/SQLite/response paths, prove byte-exact snapshot/read/corruption/retirement/durability, and measure before/after OS-level opens and wall time on the same managed endpoint-scanned large repository. Independent approval and native macOS are also absent. Do not merge/auto-merge #4792 or close #4533 based on partial improvements; no Kev/decision-model or release changes.


Published: staged inode authority across canonical deduplication (2026-11-09)

Commit 8e17075 closes a source correctness race: a private segment inode is authenticated during staging, but the canonical-collision comparator independently opens and measures its current pathname. If the staging pathname is replaced with another same-byte/different-inode file after the earlier witness check but before comparator metadata acquisition, byte equality alone previously authorized deletion of the original staged copy. Both the existing-canonical path and atomic no-replace collision path now recheck the originally authenticated temporary inode after equality, before discarding the staging copy. The checks are metadata-only, adding zero pathname opens. The new outer immutable-path comparison tracing span measures the endpoint-authorized opens; the original inner byte-read span is preserved.

A production-writer/durability-path regression stages nonempty SHA-256-addressed bytes, installs a competing identical canonical name, then uses the new production comparison span to deterministically swap the temporary inode after the initial witness check and before the comparator samples metadata. It requires a nonzero real comparison, typed refusal, intact canonical bytes and both staging file contents, retained pending ownership, and retry refusal even after the old inode is renamed back (ctime changed). The predecessor accepts and deletes the replaced pending name. This test is committed, not yet verified passing. Exact-head hosted Bazel CI is queued; macOS native is not enabled. No Cargo, no force update, no CI workflow changes.

Refs #4533, partial only. Cold first-opens remain O(distinct segments); packed/offset storage, full daemon open-path audit, managed endpoint-scanned before/after measurements, independent approval, and on-master acceptance remain outstanding. No merge/auto-merge or issue closure.

Latest source fix — canonical deduplication identity (2026-11-09)

Commit e18d295 makes canonical/pending immutable-segment equality fail closed when either pathname or opened inode changes while a potentially slow comparison runs. Previously files_equal could report equal opened bytes after a concurrent same-byte different-inode pathname swap, then authorize removal of the private staged segment. Unix checks the original pathname/fd identity before and after comparison, without extra pathname opens. A real nonempty same-byte replacement test swaps the canonical inode inside the production comparison span, demands refusal, checks exact bytes/inode and recovery. The predecessor would accept the comparison. Refs #4533; do not merge yet: the new regression and formatting are not validated on the current head, and O(1) cold first-opens and full endpoint acceptance remain outstanding.

Refs #4533

Latest source repair — failed cold open leaves healthy LRU intact (2026-11-09)

Commit 2e1b241 fixes a verified source-level cache-admission gap: after a valid ninth staged segment passed pathname metadata checks, the eight-entry pinned LRU evicted its oldest healthy handle before File::open. If that pathname disappeared or open was refused in the intervening syscall, the healthy descriptor was lost, and a subsequent read paid another endpoint-authorized open. The candidate now opens and authenticates before the old descriptor is evicted. The retained cache stays at eight entries with one transient candidate at admission. Existing inode, digest, corruption, output-clearing, and poison checks are unchanged.

The new Linux real-filesystem regression staged_failed_cold_open_preserves_eight_healthy_handles stages nine different nonempty content-addressed segments, proves eight successful opens with exact bytes, and deterministically moves the ninth file at the existing production open trace event after metadata validation but before its real syscall. It requires one observed failed open attempt, cleared output, all eight healthy handles retained, and zero further opens for the original oldest segment. Restoring the same inode then permits exact ninth-segment bytes; the predecessor evicts one handle and fails this test. The assertion is nonvacuous and uses production reader/writer behavior, not a mocked opener.

Validation hold: the patch is published but has not been proven passing by Bazel at this head. Current hosted Bazel CI #42620737107 was pending at inspection; native macOS was not requested by that workflow. No Cargo was used. Independent approval, full Linux/Windows/native verification, packed O(1) cold storage, daemon-wide open-count audit and same-managed-endpoint before/after timing remain missing. Refs #4533 only; do not merge this partial PR or close the issue based solely on green summary gates.

Latest published repair — staged checksum failure poisons preparation

Commit 76b7556 closes a fail-closed gap: a corrupt auxiliary staged segment could be opened after mutation, fail its first whole-segment SHA-256 read, yet remain cached and leave the preparation writable when the inode metadata stayed stable during the read. The reader now invalidates the entire open preparation on a failed authenticated read, clears the returned bytes and pinned handles, and refuses repeated requests without another pathname open. Neither restoring the segment bytes nor committing a valid-looking manifest can reauthorize the transaction.

The new real-filesystem regression staged_segment_cold_digest_mismatch_poison_survives_restoration deliberately corrupts a nonempty staged segment before the first pinned open, so the SHA-256 check rather than a changing-metadata check detects the error. It requires a positive file-open count, cleared output, zero retained handles after the first failed read, no retry opens, refusal after restoration, and failed write/commit with no installed active generation. This is a new published regression, not yet proven passing; the previous implementation retains a handle and therefore fails its zero-handles assertion.

Hosted Bazel CI run #42581012282 is queued on this head. No local Bazel executable/checkout is available, and no Cargo was executed.

Scope and current state

This remains a partial code-index storage improvement. It reduces redundant pathname opens and fixes integrity/durability failures, but does not yet implement packed storage or O(1) cold first-opens across distinct segments. Keep the PR unmerged and #4533 open on this PR's summary checks alone.

The PR now targets master, not draft #4837. Integration commits 4ea334e and 835bd99 retain the existing branch history and adopt the exact CI configuration already merged on master, ending at 02e3c8b28b8dfc73d6fcf9b4e36120256dbe01c0. The five preceding storage feature files were preserved byte-for-byte during integration. The resulting proposed diff has six storage source/test files and no CI, workflow, dependency, or infrastructure changes. No action was taken on #4837 itself.

Current head: 76b7556730ce493ed871f172a2c9639f04640163.

Latest source repair: final durability barriers

Commit e9b6e8efd8d7355b10a00739e7dfdd9aa7f45184 fixes a gap in Linux DurableFileBatch::sync: earlier flushed windows were checked only before the final pending window's potentially blocking fsyncs. An earlier pathname could be replaced during that last barrier and the batch could still report success.

The final sync now checks earlier identities again after the pending barriers complete. The initial fail-fast checks, error propagation, descriptor bound and failed ownership remain intact. This adds metadata checks, not pathname opens or additional fsyncs.

The new real-filesystem regression framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier:

  • Writes 33 nonempty real members, proving the first 32 barriers occurred and an intact final sync succeeds.
  • Uses the existing production fsync tracing span to deterministically rename an equal-byte, different-inode replacement over an earlier member during the final barrier. It uses a thread-local subscriber, one pending member, no sleeps for scheduling, no mocked fsync, and no new production test hook.
  • Requires one actual observed span entry, positive durability-barrier counts, typed InvalidData, exact replacement bytes, retained batch ownership, and refusal on retry before another barrier.

The preceding implementation returns success in this interleaving; the new assertion would fail it. This is a published regression, not yet an executed passing result. Existing batch and staged-reader tests are unchanged.

Existing production improvements retained

  • Eight-entry Unix/macOS staged-reader LRU with exact request validation before cache mutation; original inode identity survives eviction, pre/post-read checks reject changes, and failures clear output.
  • Linux durability reuses at most 32 duplicated writer descriptors and concurrent bounded fsync windows instead of reopening each segment. Admission is bound to authenticated writer metadata; failed writes/admissions do not leave registered deleted temporaries.
  • Existing, newly staged and generation-evidence segments retain checked SHA-256/inode witnesses, avoiding repeated digest pathname opens without accepting substituted inodes.
  • Atomic no-replace publication for immutable generation and segment names, exact-content deduplication and collision refusals; manifest hashing reuses the writer handle.
  • Unix directory identity checks avoid redundant directory opens. Pointer validation and CAS obtain authenticated bytes from one descriptor; pointer memoization reuses the persisted tempfile descriptor and rejects symlink substitution.
  • Retained reads preserve pre/post-I/O cancellation/deadline precedence, empty failure output, whole-segment SHA-256 checks, and fresh-store snapshot reload coverage.

These are source-level improvements. They are not managed-endpoint wall-time measurements and do not eliminate the first open of each distinct segment.

Validation

Prior-head hosted Bazel run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42579869168 (superseded by the published checksum repair). New-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42581012282 . No current-head passing Rust, formatting or Clippy result is claimed here.

The prior head b42f7ca396 had successful hosted repository gates, benchmark-harness checks and Dashboard at the last inspection. Its Linux fleet was queued and Clippy was still running. An earlier owner report records 18 staged-reader tests and 33 private-filesystem unit tests passing under local Bazel on macOS at that prior head; that report is not evidence for this new Linux-only regression or the current head.

Relevant generated Bazel targets remain unchanged:

bazel test //crates/tracedecay-private-fs:unit_test --test_output=all --test_arg=framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier
bazel test //crates/tracedecay-private-fs:unit_test //crates/tracedecay-private-fs:framed_log --test_output=all
bazel test //crates/tracedecay-code-index-runtime:unit_test --test_output=all --test_arg=code_index_scheduler::publication_store::staged_segment_reads
bazel build //crates/tracedecay-private-fs:rustfmt_check_2024 //crates/tracedecay-code-index-runtime:rustfmt_check_2024

Require a nonzero executed count and inspect the tested SHA. This environment has no local Bazel executable/checkout or outbound checkout DNS; no Cargo or direct Rust compiler/formatter execution was used. The available GitHub connector has workflow inspection/reruns but no native workflow-dispatch action. Native hosted macOS and current-head independent approval remain outstanding. Existing resolved inline review threads are not approving reviews.

Remaining issue acceptance

The implementation still needs the architectural packed/offset-index cutover (or a demonstrated equivalent), including every production writer, decoder, graph/lexical reader, sparse successor and retention caller. A fresh generation still has O(distinct segments) cold first-opens. Also outstanding: the complete daemon-path open-count audit (verification, restore/reseat, retention, session transcripts, SQLite temporary files and response handles), full snapshot/read/corruption/retirement/durability proofs, and before/after OS-level opens plus wall time on the same managed endpoint-scanned large repository.

Graph-specific work remains separately owned by #4796. Keep #4533 open until all acceptance is proven on master. No release, tag, production, credentials, protected Kev/decision-model changes, or external delegation.

The owner’s master synchronization subsequently moved this PR back to master at 835bd99377. Its six-file diff is confined to the original five storage paths and the added final-barrier regression. The current master cache snapshot is the second merge parent and the PR is mergeable. On this head all 18 staged-segment read tests and all 33 macOS private filesystem tests passed under Bazel. The final-barrier interleaving uses the Linux-only deferred fsync windows and awaits the hosted Linux target.

@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 60f5bcd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review November 8, 2026 00:55

Copy link
Copy Markdown
Owner Author

#4533 owner / sweep coordination (partial PR, Refs #4533): Please do not merge or enable auto-merge based solely on green workflow gates. This head needs a nonzero Bazel Rust regression run, relevant native hosted checks and an independent correctness review first. The full packed-generation storage / large-repository endpoint-security acceptance remains separate, so #4533 must stay open even if this reviewed partial lands. PR #4796 owns the related graph-pass path; avoid overlapping edits. No release or Kev/decision-model changes are authorized in this lane.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure.

The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader.

Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used.

The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements.

Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes.

Keep the concurrent formatting commit in the branch history. Rehome and
extend its production-reader regression to cover in-place mutation,
transaction poisoning, failed reads, and escaped-reader cleanup.

Refs #4533
Keep the first-open file identity for the preparation even after its
bounded read cache evicts the descriptor. Refuse a replaced pathname
before another open, including across the cache's eviction boundary.
Prove the refusal with a real nine-segment regression.

Refs #4533
Exercise a real staged generation's whole and repeated range reads,
verify the content digest and exact bytes, then publish and cold-decode
its durable snapshot without changing the production storage format.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits 262e340 (original pin identity, pre-open failure output hygiene, mutation/commit tests), b850dcb (persist first-open identity across LRU eviction + poisoned no-reopen regression), 18269c6 (real staged whole/range bytes through successful commit and cold snapshot reload regression) are now on fix/4533-staged-segment-read-opens. Tests are added but not yet executed; hosted Bazel CI for the current tip is pending, macOS native CI is not running by default, and independent review is still missing. Please do not merge/auto-merge this partial PR solely on default green CI. Fresh audit finding: stage_segment still creates one file per new segment; verify_prepared_segments invokes state_digest_file for each distinct segment (another per-file open); on Linux, DurableFileBatch::sync calls sync_file_at for every written member, which uses OpenOptions::open again per file. The full #4533 packed-storage cutover must remove those O(segments) first opens and prove durability/retention as well as repeated read behavior. PR #4796 independently owns graph-read instrumentation; do not merge overlapping change here. Refs #4533; keep issue open. No release, production, Kev/decision-model or restricted actions.

Quote the pathname API in documentation and retain exact empty-buffer
regressions using equality assertions required by the pinned Clippy policy.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head 103101b6 and fixed its actionable Clippy failures in commit c76238b, appended non-force with expected remote head to the existing issue-specific PR. The blocking Rust Clippy errors were doc_markdown (unquoted Handle::from_path in publication_store.rs:2162) and two assert_is_empty findings in the staged-reader regressions (staged_segment_reads.rs:240,334). Both equality assertions continue proving the output buffer is exactly empty; no test was removed, skipped or weakened. The earlier Linux fleet failure (UNAVAILABLE: io exception building tracedecay-daemon-service) is a separate remote execution failure, not evidence of a source fix. New hosted run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42441637324 — not yet green at this writing. Native macOS is skipped in default CI; independent review and nonzero focused Bazel execution remain required. The full O(1)-first-open packed-storage cutover, before/after macOS endpoint-security measurements and index/restore/snapshot acceptance remain open. No duplicates, merging, release, Kev work or infrastructure actions. Keep #4533 open and do not auto-merge #4792 on CI green alone; coordinate with #4796 graph reader work.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner — production fix published at 3286852 on this PR. Appended with force=false and expected remote head c76238bd05e9eb356ee4963010f1036f62f88ba4; concurrent Clippy repairs are preserved. No duplicate PR or merge.

Changed crates/tracedecay-private-fs/src/framed_log.rs: Linux durability batches retain at most 32 duplicated writer descriptors, flush full windows concurrently, and use those descriptors rather than reopening every pathname. Earlier windows retain original metadata so missing/replaced/modified members still refuse final sync. Pending path/descriptor identity is checked before and after fsync. A failed full window retains its handles and cannot be marked successful or admit another descriptor. Non-Linux durability, existing integration tests, checksums, format and publication ordering are unchanged.

Added seven real-filesystem regressions in src/framed_log_batch_tests.rs, wired into the existing Bazel unit target: positive-control kernel IN_OPEN counts across 67 files, bounded descriptors, exact byte readback, nonzero early/final barrier counts, mismatched writer/path refusal, pending/early-flushed replacement, missing/modified early files, and failed-window ownership/retry. Expected open count is 67 writer opens with no durability reopens, NOT an executed result. Existing member-only and concurrent slow-fsync integration tests remain intact.

Verified generated targets: //crates/tracedecay-private-fs:unit_test, :framed_log, and :rustfmt_check_2024; no BUILD/dependency or workflow changes. Root diff reviewed against the exact parent: changes are limited to the batch implementation/imports/test wiring and the new tests. #4796's graph-reader lane and existing staged-read code were not overwritten.

Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used.

This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work.

Reuse duplicated writer descriptors for Linux durability windows instead
of reopening every segment. Preserve concurrent member fsyncs and refuse
missing, replaced, or mutated members, including earlier flushed windows.

Add kernel-observed open-count, barrier, ownership, and failure regressions.
Refs #4533

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

Published the direct GitHub source repair at 4d0ad18, appended with force=false and expected remote head 32868523be33a85fb091feebc1d2b11c1b346143. The concurrent framed_log.rs durability commit and prior Clippy fixes are preserved. No duplicate PR or merge.

The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not.

Added staged_segment_rejected_reads_preserve_full_cache_and_recency using the real preparation/writer/reader: nine staged files, eight positively exercised cached handles, overflow/out-of-bounds/wrong-size/missing-path refusals, empty output and unchanged cache on each refusal, then exact bytes and open counts 9/9/10 to prove eviction order. All five pre-existing staged-reader tests are unchanged; no source-shape acceptance or weakened assertions.

Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval.

The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533.

@ScriptedAlchemy ScriptedAlchemy changed the title perf(code-index): pin staged segment reads within preparation perf(code-index): reuse staged read and durability handles Nov 8, 2026
Validate ranges and named extents before changing staged-reader LRU state.
Reuse admitted metadata across the open, retaining replacement checks.
Add a real full-cache regression for cold refusals and eviction order.

Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the
same-inode durability regression without changing its assertions.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner — hosted gate repaired on the existing PR (no merge). At 4d0ad18, Repository gates failed its Bazel rustfmt_check_2024 on one line in framed_log_batch_tests.rs:214. I committed the exact required formatting correction as 688f3a6 and advanced fix/4533-staged-segment-read-opens by expected-head non-force update. No test assertion or production semantics changed in this repair. New-head hosted CI is pending; macOS is skipped in the standard workflow and no independent review/nonzero Rust test result is established. Maintenance sweeps: please preserve the no-merge/auto-merge hold. Refs #4533 remains partial: packed storage, broad open audit and endpoint-scanned before/after verification are unfinished. No changes to #4796, releases, infrastructure or Kev.

Publish missing canonical names with atomic no-replace semantics. If another
publisher wins, verify byte identity before deduplicating and preserve any
conflicting immutable content. Exercise the production path for conflicting,
identical and absent destinations.

Refs #4533
On Unix use the directory device/inode identity directly when preparing and
validating a publication. Preserve existing Windows handle identity checks
and reject substituted directories before commit. Add a real prepared
publication regression for changed and restored directory identities.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:

  • 70188c58: atomic no-replace for staged immutable segment names; verified collision/dedupe/absence paths in a new real-filesystem regression. Prevents a concurrent publisher from being overwritten by a stale missing-path observation.
  • 7184d12e: on Unix/macOS replace six per-generation directory Handle::from_path opens (three on prepare and three on commit validation) with dev/inode metadata identity comparisons; preserve Windows checks. Added a real staged-publication substitution/restore regression exercising compare-and-swap refusal.

Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work.

Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it.

Copy link
Copy Markdown
Owner Author

#4533 owner-lane / sweep and PR-maintenance coordination — published, partial, HOLD MERGE. Preserved concurrent work on the existing master-targeting #4792 and published two guarded nonforce commits: 9ea8a39, exactly the five hosted Bazel formatting hunks without weakening assertions, and 61394f7, wiring retained-history descriptor release into normal exclusive-lock generation retention. The new real two-generation Linux regression requires a positive cached descriptor, then zero process file descriptors to the unlinked inode, stale-read refusal and active-snapshot correctness. Hosted CI 42712609686 is queued; no current-head passing Rust tests or independent approval. The retained-inode review finding has a response but is not marked resolved. Please coordinate touching the seven updated files with this owner lane, keep PR unmerged and issue open. Packed O(1) cold first-opens, full daemon audit, managed endpoint before/after measurements, and independent approval remain outstanding. No Cargo, forced history, duplicate PR, external delegation, Kev/draft #4776 or release #4763 modifications.

Apply all 21 exact Rustfmt hunks emitted by hosted Repository gates
on 61394f7 across six changed production/test files. No semantic or
assertion changes. Keep the issue PR targeting master, Refs #4533.
A historical page read still opened the active pointer for every range,
even when its segment was cached. Pin the authenticated Unix pointer inode
in the existing digest memo, rehash its bytes through that descriptor and
verify path/fd identity before and after reads. Atomic publication changes
fall back to cold verification; same-inode mutations refuse corruption.

A Linux native inotify production-history test requires one cold pointer
open and zero more across 17 exact-byte reads and a cloned store, with a
positive kernel-open control. Refs #4533; partial, do not merge.

Copy link
Copy Markdown
Owner Author

#4533 owner-lane checkpoint — published source and Bazel CI repair (Refs #4533; partial; HOLD MERGE).

Two verified non-force, expected-head fast-forward commits on the existing master-targeting PR:

  • 43ca411 applies all 21 exact formatting hunks from hosted Repository gates #42712609686, across six files. The eight repository script tests passed on the old head; rustfmt was red. No runtime behavior or assertions were changed in this commit.
  • fba00d6 fixes a separate per-page endpoint-open source: read_retained_partitioned_segment checked the active pointer for every historical page, and read_publication_pointer_with_bytes unconditionally issued a new pathname File::open even with a warm pointer memo. On Unix the canonical pointer memo now retains its authenticated descriptor, re-reads/rehashes the current bytes on that same FD, and checks opened and named inode identity before/after. Changed pointer inode falls back to the existing full cold path; mutations on a pinned inode refuse rather than trusting timestamp/size. This also preserves read-after-publication by borrowing the persisted tempfile descriptor via try_clone without an extra pathname open.

The existing real-production-publication Linux inotify regression is strengthened: one kernel open for cold pointer authentication, zero additional pointer opens for 17 historical whole/range reads and a cloned store, plus a positive independently observed pointer open() control; segment-byte equality, digest, one-open segment cache, inode-substitution refusal and empty failure output remain. These assertions are committed, not yet verified passing. Exact-head hosted Bazel CI #42716161880 is queued; native macOS is skipped.

Additional audited blocker: read_retained_partitioned_segment still calls try_acquire_code_generation_store_read_lock for every range, and code_index_generations/locking.rs opens STORE_LOCK_FILE on each acquisition. Thus removing repeated pointer/segment opens does not establish O(1) total endpoint-authorized opens; the read-lock authority must be redesigned without losing cross-process exclusion, cancellation, or retention safety. Distinct sealed segments also still cold-open O(segments). This remains a partial storage change, not a packed/offset-index cutover or managed-endpoint performance result.

Independent approval, current-head nonzero Bazel checks, full daemon paths (restore/reseat, verification, retention, transcripts, SQLite temp, response handles), snapshot/retirement proofs, and same managed-host syscall-open/latency before-after are pending. Coordinate further edits to the existing issue branch; do not merge or close #4533. No Cargo, force/rewrite, test relaxation, delegated engineering, decision-model/Kev/#4776 or release/#4763 changes.

devin-ai-integration[bot]

This comment was marked as resolved.

Clear the shared pointer memo before the destructive reset, retaining its
mutex through deletion. Later reset failures must not retain an unlinked
pointer descriptor across store clones.

Add a real generation publication, contended-reset preservation,
post-unlink failure and fresh-store recovery regression. Linux observes
actual held inode descriptors before and after reset.

Refs #4533

Copy link
Copy Markdown
Owner Author

Review repair published: 0accee86, appended to fba00d6 via exact expected-head, nonforce update. Two files only: pointer-memo reset ordering in publication_store.rs and the new owning-runtime publication_reset.rs regression. This addresses the independently reported deleted-pointer descriptor leak on reset failure. The memo releases its descriptor before unlink and stays locked through destructive reset; the test covers lock-contention preservation, a real post-unlink failure, positive/zero Linux inode-descriptor observations across clones, and fresh-store exact snapshot recovery.

Hosted Bazel CI #42723182058 is queued for this head. No current-head passing Rust/format/Clippy result or independent approval is claimed. The review thread has been answered and is left for independent re-review. Preserve this head unless there is an actionable source or CI correction; do not churn pending validation.

Refs #4533 only. The distinct-segment packed-storage cutover, per-page generation-lock opens, complete daemon open-count audit, and managed-endpoint measurements remain outstanding. Keep this PR unmerged and #4533 open. No Cargo, force/rewrite, test weakening, infrastructure, protected Kev/decision-model or release changes.

Exercise the production metadata reader for vanished/directory path errors,
retain sibling-test visibility of the bounded descriptor cache, and supply
the absent optional retirement callback in the isolated text-artifact test.
Apply the exact hosted Rustfmt hunk to the failed-reset regression.

These repair compilation and formatting without removing assertions or
changing storage behavior. Refs #4533
The retained cache's sibling unit test reads its bounded length. Match the
new scheduler-private field visibility on its element type to avoid a
private-interfaces warning under Bazel Clippy -Dwarnings.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner/sweep coordination — published compiler/format repairs; Refs #4533 only. New head ec15c307 after 63a01e7, both with exact remote parents and nonforce expected-head ref updates. These repair three confirmed compiler-failure families from Bazel #42716161880 and one exact Rustfmt hunk from #42723182058, with no test deletion or weakened assertions. Current-head Bazel CI #42726694390 queued, NOT green. Please preserve this head for verification; coordinate overlapping maintenance edits. Packed/O(1) distinct cold opens and per-page generation lock overhead, daemon-wide open-count audit, managed endpoint measurements and independent approval remain outstanding, so do NOT merge #4792 or close #4533. No Cargo, protected Kev/#4776, release/#4763 or infrastructure action.

Retain the missing-generation and non-file refusal assertions while avoiding
expect_err's Debug bound on the metadata success type. Remove a Clippy
redundant closure in the live retained-segment descriptor regression.

Refs #4533
Keep a per-store, clone-shared Unix lock descriptor while taking and dropping
an actual kernel shared lease for each historical page. Authenticate the lock
pathname inode before and after acquisition; fail closed on replacement and
retry rather than block a contending in-process reader. The Linux production
read regression now counts actual lock-file opens across 17 reads and a clone,
including an independent positive event-observer control.

Refs #4533
Exercise real Unix kernel shared and exclusive leases across cached reads,
including same-process contention, explicit release, recovery and a replaced
lock pathname. A stale cached descriptor must refuse the new inode.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner/sweep coordination — published partial fixes (HOLD MERGE). Branch head 30747aaa, exact-parent expected-head nonforce chain from ec15c30. 46c01415 repairs confirmed hosted Bazel Clippy/test compilation without weakening error assertions. a840b5c9 removes per-historical-page generation-lock open() with a clone-shared cached descriptor while reacquiring/releasing the real kernel read lock each time; inode changes fail closed. 30747aa adds real exclusive-writer and stale-inode refusal tests. Linux inotify regression requires 1 cold lock-file open / 17 reads, 0 more for a clone, plus independent open control; existing exact segment/pointer/snapshot verification remains.

Exact-head hosted Bazel CI #42737264260 queued, NOT passed; native macOS skipped. Related PRs #4796 (graph), #4789 (sparse evidence), and #4846 (graph proof) have no direct changed-path overlap with this lock/cache work. Do not merge or close: O(distinct cold segment) opens still requires packed/offset storage or equivalent, full daemon open-path audit, managed-endpoint measured latency and independent approval. No Cargo, force/rewrite, protections bypass, protected Kev/decision-model (#4776) or release (#4763) changes. Refs #4533.

Authenticate the lock-file directory entry with lstat rather than
following a symlink to a retained descriptor. A same-inode symlink
replacement must not silently authorize historical reads under an
old lock authority. Retain the one-open cache and kernel fencing.

Extend the real Unix kernel regression with a positive same-inode
symlink control, refusal, exact-inode restoration, and writer exclusion.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner lane / sweep coordination — published narrow lock-authority correctness fix (Refs #4533; partial, HOLD MERGE).

Published 53b765cf as an exact-parent, expected-head, non-force fast-forward from 30747aaa on existing PR #4792 targeting master. One owned file changed: crates/tracedecay-code-index-retention/src/code_index_generations/locking.rs.

Confirmed source flaw: The new reused historical-read generation-lock descriptor compared std::fs::metadata(path) with file.metadata(). Because the pathname check followed symlinks, an attacker or stale authority could rename the lock file and replace its directory entry with a symlink back to the retained old inode; the check accepted that substitute and granted a kernel shared lock on the retired authority. The implementation now uses symlink_metadata, rejecting that alias with no new open(), no weakened fencing, and no changed store format.

Nonvacuous regression: Existing real Unix kernel-lock test now creates both a different-inode replacement and a same-inode symlink replacement; it proves the latter resolves to the exact cached dev/ino, requires typed refusal, restores the original inode, reacquires a real shared lock, and proves an exclusive writer is still excluded. The prior head fails the symlink-refusal assertion. Source and regression are committed; they have not been shown passing under Bazel on this SHA.

Current-head hosted Bazel run #42744217678 is queued, native macOS not selected; independent approval and nonzero current-head test/Clippy evidence remain required. The architectural O(distinct segments) cold-open bound, full daemon-wide OS open audit, and same-managed-endpoint latency comparisons remain outstanding. Do not merge/auto-merge #4792 or close #4533. Preserve this head and coordinate any overlapping sweep edits. All Kev/decision-model work, including #4776, remains DRAFT; release #4763 read-only. No Cargo, test weakening, external delegation, force updates, releases or infrastructure changes.

Reject both existing-target and dangling symlinks before pointer CAS or
rename. The cold authenticated pointer recheck also uses lstat to forbid
a symlink substitution to the same opened inode, without another open.

Exercise real symlink pathname refusal, untouched target/link identity,
and exact-byte ordinary publication in the owning runtime tests.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner/sweep coordination: published narrow pointer-slot safety fix (Refs #4533, HOLD MERGE). 20c5a356 fast-forwarded the existing master-targeting PR from 53b765c with exact expected-head and no force. The commit refuses dangling/matching-byte symlink pointer slots during CAS and closes the cold-authentication final symlink-follow gap; it adds a real Unix successful/denied CAS regression. Hosted Bazel CI #42747934032 is queued; tests are not yet passing. PR #4796 remains the separate graph open path lane—no duplicate #4533 PR or overlap edits. Keep #4792 unmerged until independently reviewed, nonzero CI plus O(distinct segments)/managed-endpoint acceptance, and final-V2 snapshot/retirement proofs. Do not merge Kev/decision-model work (including #4776) or mutate release #4763.

Fail closed when the final cold publication-pointer witness detects a
replacement inode, symlink alias or changed bytes. Previously the reader
skipped memo installation but still returned the stale parsed pointer.

Exercise a real opened inode against same-byte replacement and symlink
alias, followed by valid restoration, without an additional production
pathname open.

Refs #4533
A real same-inode rename updates Unix ctime. Refresh that metadata for
the positive restoration control while preserving the same open descriptor;
keep the byte-identical inode replacement and symlink refusal assertions.

Refs #4533
Refuse a cold pointer after a pathname rotation without returning stale
bytes. Treat legitimate concurrent replacement or unlink as a retryable
CAS conflict, not a destructive corruption reset; retain corruption for
symlinks and in-place byte or inode metadata mutation.

Exercise missing path, same-byte new inode, symlink alias, and restoration
through real descriptor metadata with no additional production open.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner/sweep coordination (Refs #4533, partial): reviewed the current root and relevant nested AGENTS.md inventory, complete issue/comments, master reader, related #4796/#4547, review threads and CI. Published ebb9f1d → fcbe281 → e86a4e5 on the existing PR via exact-parent expected-head nonforce updates. Cold pointer final-witness failure now refuses rather than returns stale bytes; legitimate name replacement/missing path is retryable CAS (not destructive reset), while symlinks/in-place corruption are refused. Added real descriptor/inode/same-byte/symlink/restoration assertions; restored ctime is correctly refreshed. Hosted Bazel current-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42752854848 (queued; no pass claimed). Please do not rebase, duplicate, merge, auto-merge, or close in broad sweeps. Packed/offset storage, daemon-wide open-count audit, managed-endpoint results and independent approval remain required. Protected Kev/#4776 and release #4763 untouched.

Apply the seven exact formatting hunks emitted by the current-head
Rustfmt gate for retention and runtime, preserving all behavior and
nonzero assertions.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner/sweep coordination — published hosted formatting repair, Refs #4533 only (HOLD MERGE).

Published 33df990 on existing #4792, exact child of e86a4e5 with a non-force expected_sha fast-forward; verified the remote PR head afterward. This applies the seven exact Rustfmt hunks from hosted Bazel run #42752854848 across only retention and runtime source (import ordering, two assertions, struct formatting, pointer identity conditional). The previous run passed eight repository script tests but its //:rustfmt_check failed; no behavior, assertions, CI gates, or protected branches were changed.

Current-head Bazel CI #42758341694 is pending and not a passing test result; native macOS is skipped by this workflow. The open issue remains O(distinct segments) cold first-opens, final-V2 packed/offset-or-equivalent storage, daemon-wide per-operation opens, managed-endpoint before/after latency, and independent approval. Preserve this head in other sweep tasks; no rebase/duplicate/merge/auto-merge/issue closure. Kev/decision-model work including #4776 stays DRAFT; release #4763 read-only. No Cargo, external delegation, or infrastructure actions.

Recheck every pending member after all fsync workers join, before a
window can succeed or release its retained descriptors. Per-member
post-barrier checks alone miss changes while another worker finishes.

Cover same-byte inode replacement during final sync and same-inode
mutation during rollover with real fsyncs and deterministic, test-only
completion handshakes. Preserve failed admission and exact recovery.

Refs #4533

Copy link
Copy Markdown
Owner Author

Pushed b852ca0 (fix(private-fs): revalidate completed durability windows) with an expected-head, non-force fast-forward from 33df990. Remote branch head verified.

Found a separate pending-window durability race: an early member could change after its own fsync and identity checks while the concurrent window was still running. sync_pending previously returned the joined worker result without re-authenticating the entire pending window. Final sync could report success, or rollover could promote changed members and release their descriptors.

The fix checks every pending descriptor and pathname after all workers join, before success or rollover promotion. It adds no pathname reopens or fsync calls and preserves the existing bounded concurrency window and failed-admission ownership.

Added two owning-crate regressions in framed_log_window_tests.rs: identical-byte inode replacement after a member completes during final sync, and same-inode/same-length mutation after a member completes during full-window rollover. Both use real filesystem operations and real fsyncs with a deterministic completion handshake in the existing crate-private, cfg(test) fixture. Assertions cover positive barrier observation, typed InvalidData, retained pending handles, refused retry/new-member admission, and exact fresh-batch recovery for the replacement case. No new downstream public test API or dependencies.

Validation: reviewed the final commit diff and existing Bazel unit-test wiring. No local Bazel/Rust test execution was available. Exact-head hosted CI run 42763318601 is queued: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42763318601. The regressions are committed, not yet established passing. No merge or issue closure; this does not claim full #4533 acceptance.

Apply the three exact rustfmt hunks reported by current-head Bazel
repository gates for the durability window regressions. Preserve all
production logic, test assertions and deterministic interleavings.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — current-head owner checkpoint (PARTIAL / HOLD MERGE).

Published 108b39fa on this existing PR with an exact-parent, expected-head non-force fast-forward from b852ca00; no duplicate PR, rewrite, or peer edits discarded. Hosted Bazel CI #42763318601 confirmed //crates/tracedecay-private-fs:rustfmt_check_2024 failed on exactly three formatting hunks in the new fsync-window regression; the eight repository script tests passed. The commit applies only those three emitted hunks. No assertions, test flow or production behavior changed. New-head CI #42768671504 is queued, not green. Native macOS remains skipped.

Read root AGENTS.md/current issue (no comments), head and master storage/retention/test implementation, related #4532 PR #4796, all visible #4792 review submissions and threads, and hosted CI. The existing #4792 is the appropriate issue PR; no competing #4533 PR found. One cancellation-order review thread remains independently unresolved (source correction present); no approving independent review is recorded.

Merge hold regardless of CI: #4533's fundamental 18k+/73k+ distinct-segment cold-open problem remains O(segments); the existing bounded reader cache and reuse of lock descriptors do not prove O(1) cold first-opens. Packed/offset-indexed storage or another demonstrated bound, a complete daemon-wide per-operation open audit (including restore/reseat, retention, transcript, SQLite temp and response paths), byte-exact final-V2 snapshot/corruption/retirement proof, same-machine managed-endpoint before/after open counts and latency, and independent approval remain outstanding. Do not merge/auto-merge or close #4533. Kev/decision-model including draft #4776 remains protected; release #4763 remains read-only. No Cargo, Kody/Copilot/external delegation, infrastructure or release action.

Refuse symlinked generation names and inode or metadata changes across
full and metadata-only retention scans. Keep the same single pathname
open while validating the original descriptor and directory entry.

Add owning-crate real-file regressions for an alias and a same-byte
foreign-inode rotation during both verification modes, plus a healthy
full-hash control. Neither path requires additional open calls.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published code-index retention correctness repair on the existing master-targeting PR, preserving peer work.

Exact fast-forward: 60f5bcdb from 108b39fa via non-force expected_sha; remote PR head verified.

Confirmed source defect: The retention metadata scanner accepted a symlink under a generation-shaped pathname: generation_file_name follows is_file(), then read_generation_metadata follows the symlink. Worse, in MetadataOnly mode it derives the identity digest from that pathname, so a foreign inode can masquerade as a valid indexed generation without byte authentication. A same-byte inode replacement during the read could also leave a mismatched manifest/filename observation.

Fix: read_generation_metadata refuses nonregular pathname entries, authenticates the opened inode against the enumerated entry on Unix, and rechecks the original pathname and descriptor after either full hashing or metadata-only scanning. The existing single File::open remains the only pathname open; extra witnesses are metadata syscalls, not opens. No legacy format or API path introduced.

Falsifiable owning-crate regressions: A genuine content-addressed generation plus a symlink alias is rejected in both modes while the original still decodes with its exact full SHA-256; a same-byte, different-inode atomic replacement at the third real cancellation checkpoint (after the first descriptor read) must fail in both modes, retaining replacement bytes. Old code accepts these reads. These are published, not yet executed; no local Bazel checkout/DNS. New-head hosted CI #42777818917 is pending. Previous-head #42768671504 passed Repository gates and Dashboard but native/Clippy jobs were superseded by this push, not green.

HOLD MERGE / PARTIAL: The primary O(distinct sealed segments) cold first-open problem is still unsolved: packed/offset storage or a demonstrated equivalent, full daemon open-path audit, managed endpoint-security before/after measurements, snapshot/retention proof and independent approval are outstanding. Please preserve this exact branch in general PR sweeps; do not duplicate, auto-merge, merge, rewrite or close #4533. Kev/decision-model (including #4776) stays draft, #4763 release read-only. No Cargo, external delegation or infrastructure changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code-index: store and read the index in few files so endpoint security open() cost does not dominate

1 participant