Skip to content

fix(sqlite): publish complete stores and recover interrupted initialization - #4805

Open
ScriptedAlchemy wants to merge 64 commits into
masterfrom
fix/4513-reserve-sqlite-sidecars-20261108
Open

ScriptedAlchemy wants to merge 64 commits into
masterfrom
fix/4513-reserve-sqlite-sidecars-20261108

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Refs #4513. The owner’s independent review and merge hold remain in effect. Keep the issue open for observations on master and longer-running production use.

On d8f057b013, all 1,045 tests in the native Linux ARM64 recovery suites passed as an ordinary user on ext4 (456 SQLite runtime, 554 runtime core, 35 private filesystem). An additional Linux sessions test inspected /proc/self/fd and verified the real SQLite reader remains open until its last clone closes. The native macOS suites passed 1,693 tests. The canonical sidecar namespace review is still open.

The subsequent master merge f2d1ab29f3 changes only the proximity advisory in tracedecay-application. All SQLite recovery, attachment, session and test files match the verified d8f057b013 tree.

Initialization, interrupted work and reset

The registered store initializes SQLite in a private same-filesystem sibling directory under the existing durable-directory lease. Its own SQLite writer stamps the application ID and checks for a complete WAL checkpoint. The runtime then joins its workers and passes the original main and sidecar creation handles to the publisher. The publisher never reopens the staged SQLite database. It validates the full family, syncs owned files, and may clear the disposable SHM index only through its original descriptor. A nonempty WAL or rollback journal prevents publication.

For a fresh family, already-open source and destination directories anchor no-replace moves. Three exclusive empty reservations move to the canonical WAL, SHM and rollback-journal names before the completed main moves to its ordinary SQLite name. Live handles pin the accepted file generations through publication. The published runtime checks the main generation when it attaches. On normal completion the initializer clears its preparation receipt and removes the empty staging directory. The original coordination inode stays in place.

The stable lease temporarily records the four prepared file identities and the SHA-256 of the synchronized main file. Publication verifies that digest before and after the no-replace main move and during reconciliation of a published main. Prepared content review and owner repair Recovery and admission depend on the recorded state.

  • An empty unprepared stage can be retired or reused. A handled schema error or cancellation that reaches worker shutdown preserves any committed installation steps. The initializer joins the workers, captures the original file identities and digests, and moves those same four files under a fresh owner-private recovery directory. No database bytes change during this transfer. The existing lease associates the recovery directory with the verified shard, incarnation and physical locator. A later process checks the whole family before moving any member back, then resumes schema installation using the validated handles. Success removes the empty recovery and staging directories. Handled failure and retry review
  • If the process dies during this transfer, restart checks for one instance of each member across both directories. Members already in the protected recovery directory remain provable. A member still in the active staging directory additionally requires a kernel allocation generation. When all four files moved back and the empty recovery directory was retired, the same receipt can still establish a recoverable family on a supporting filesystem. A changed payload, a missing or duplicated member, an unrecognized receipt or locator mismatch preserves the files and refuses attachment. A process death before the first complete receipt, including during a worker start or during schema writes, may still require offline reconciliation. Neither cleanup nor retry unlinks unproven regular-file names.
  • After durable preparation, a writable restart can complete the remaining no-replace moves only when the filesystem supplies a durable allocation generation for every member. It checks each member’s identity and the reservations’ emptiness. If the canonical main already moved but the receipt remains, it verifies the main and the three sidecars before releasing the receipt. Read-only attachment and offline reset refuse a pending receipt, including one whose stage is missing.
  • The Linux check uses the existing inode-generation authority where the filesystem supports it. Darwin uses a nonzero file generation when available. On filesystems returning no reliable generation, including the tested APFS configuration, and on Windows, a cold, unfinished preparation with members in the active staging namespace stays intact and needs offline reconciliation. A fully sheltered handled failure can resume after restart on those same filesystems. Normal uninterrupted publication retains live file handles. Read-only and read-write attachment to a completed store do not need durable allocation generations.
  • Existing databases with application tables in their main file or TraceDecay’s publisher marker use regular SQLite recovery. For an unmarked first schema present only in a live WAL, read-only admission checks SQLite’s existing shared index without checkpointing. An unmarked header-only main with a detached WAL and no live owner cannot establish the WAL’s ownership. Admission preserves the files. The SQLite 3.46.1 reviewer report and bundled library are distinct environments. Compatibility review

Authorized offline scoped reset and full-profile wipe coordinate with the existing initialization leases and validate expected file types. Both use the graph owner’s actual verification marker and spill paths. The full wipe checks all three top-level families before its first deletion. Unrelated paths and persistent coordination locks survive.

Review boundaries

The two recent staging findings concerned unlinking unverified leaf names and reopening or publishing a replacement database through a mutable source pathname. Failed cleanup now removes only an empty directory. The publishing code moves checked members through the retained directory capabilities, keeps their file handles live, and verifies the moved generations. Windows pins the directories against delete-sharing while resolving their paths for its native rename. Fresh publication requires same-filesystem no-replace rename. Failed staging cleanup review · Publication path review

SQLite opens its live schema-worker files and later its canonical family through its platform VFS. The store parent, private staging namespace and lease-governed recovery directory must be trusted against noncooperating writers that can replace entries during active SQLite operations or between a leaf identity check and a name-based move. A no-replace move protects its destination and subsequent checks detect mismatches. POSIX does not provide a conditional source rename by an expected inode. In particular there is still an interval from the last canonical sidecar identity check through the first canonical SQLite VFS open, followed by ordinary SQLite use. The original P1 remains open for independent review. The prospective physical factory now retains the original reservation across both writer and reader startup. On failure it returns the raw identity and original file set only after worker references release. The registered initializer moves those verified members back into recovery before returning, for both fresh and restored families. Partial creation and an unsettled or lost generation still require reconciliation. The repository factory and all five direct test callers use this same initialization entry point; the wrapper that discarded returned handles has been removed. Initialization API review Worker startup review The verified-reader POSIX lock change from #4803 is now integrated. It covers SQLite’s main file on immutable reads. Canonical sidecar P1 · Verified-reader PR

The native SQLite open interface accepts the canonical filename and a VFS. Its main and journal file controls expose SQLite’s already-open files. The VFS also manages WAL shared-memory mapping and locks. Keeping the staged connection open while moving the database is unsupported because its journal and WAL names remain tied to the original pathname. Admission already refuses occupied, missing, duplicate or changed family members when their identity can be checked. A successful check cannot establish future control over the directory by another process with the same OS account. Under this threat model a stricter permission check or advisory lease cannot establish the missing boundary. Supporting such a deployment requires a separately protected storage namespace or a different product contract for file access. SQLite open API · File controls · VFS and SHM methods · SQLite on renaming an open database

Existing physical attachments on Darwin and other Unix systems without Linux O_PATH now keep the main inode alive through a shared read-only SQLite VFS connection. Its identity is checked on both sides of SQLite’s HAS_MOVED result. The VFS coordinates the actual descriptor close with other local SQLite connections, including when reader startup fails or a lost route tears down its workers. The private creation descriptors still belong to the unpublished initialization family. Linux retains its O_PATH capability and Windows its existing file-sharing handle. Darwin attachment lifecycle review

Verification

  • On d8f057b, native macOS executed 449 rusqlite-runtime, 548 runtime-core, 33 private-fs and 663 sessions tests. All 1,693 passed and the aggregate Bazel client exited successfully. This includes the independent-writer subprocess probes for both verified-reader release and physical attachment teardown, and foreign SQLite reads through the sessions crate. The sessions last-clone descriptor-count test uses Linux procfs. Repository rustfmt and Clippy passed. Merge integrating #4803
  • On 10e09e6, all 448 rusqlite-runtime cases ran on native macOS. The unchanged 548 runtime-core and 33 private-fs cases passed from the Bazel cache, for 1,029 total. Repository rustfmt and Clippy passed. The new physical-attachment journey verified another local writer’s exclusion using fresh child SQLite processes. It covered read-only shutdown, rejected reader startup, and real writer/reader teardown after a route was lost. Foreign writes and journal-mode changes remained blocked until the transaction ended. The committed value survived rollback of the later transaction and the database passed integrity_check. The native TypeScript SDK transport journey also ingested a native session, restarted the real daemon, and verified its message, fact and refresh identity on the same production implementation. Unix regression import review
  • On 10e09e6, native Linux ARM64 running as an ordinary user on generation-backed ext4 executed all 455 rusqlite-runtime, 554 runtime-core and 35 private-fs tests. All 1,044 passed and the aggregate Bazel client exited successfully. On the earlier 622aa10, all 1,043 tests passed on Docker overlayfs without allocation generations, also with a successful Bazel exit.
  • In the registered-schema regression, two independent child-process runs commit a real row before cancellation or a failing SQL statement. The journey also rejects a real writer admission before the first install and again on a restored family. The next process completes initialization from the original row and verifies it through an ordinary read-only SQLite connection. A separate isolated SQLite reader returns SQLITE_NOMEM only after writer startup, then all handles return and a subsequent normal attachment succeeds. Separate process exits exercise partial transfers, fully restored files, and removal of the empty recovery directory. The negative cases preserve all file contents and prevent admission.
  • On 622aa10, the native macOS foreground daemon and independent system sqlite3 reader verified eight exact committed session messages across two clean daemon exits and a restart. The external reader completed five, seven and eight reads during three separate ingest requests. It checked committed content and integrity throughout. The isolated profile was deleted at the end.
  • The Linux PR workflow and the focused Windows dispatch both target d8f057b. Linux PR CI · Native Windows dispatch

At the previous reset implementation on 945dde8, all 13 scoped reset/restart journeys and all five scoped CLI unit tests passed natively on macOS. An isolated full wipe refused a held initializer and a wrong-kind marker before deleting anything. After those conditions were cleared, it removed both graph families and preserved a foreign directory and both coordination locks. Preceding Linux reset evidence

On the older 06f75fb head, native Windows ran 424 rusqlite and 509 runtime-core tests successfully. The hosted macOS job for 2ff2385 failed DNS resolution before Bazel downloaded. Previous native Windows job · Hosted macOS setup failure

The separate owner hold on the Kev/decision-model draft remains. #4776

@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b1f093e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Copy Markdown
Owner Author

Review/validation hold: This PR is marked ready solely so the existing PR CI can execute on its exact head. Please do not merge until nonzero hosted Bazel tests, native macOS/Windows SQLite sidecar and locking behavior, and the preserved-failure/retry tradeoff are reviewed. The default PR workflow skips native macOS; a skipped check is not platform acceptance. The cross-process replacement and initial sidecar refusal tests are intended to be exercised by //crates/tracedecay-rusqlite-runtime:unit_test on the current commit. The broader #4513 acceptance remains open.

Copy link
Copy Markdown
Owner Author

Refs #4513 — follow-up fix published on this same issue-specific PR at 1738c8e, fast-forwarded from exact prior head 92d53ff3f0de without force, three owned files only.

New data-loss path on retry: create_new creates the empty main DB before exclusively reserving -wal/-shm/-journal. If any sidecar is occupied, it refuses without deleting foreign files, but leaves the empty sessions.db. A later existing-mode attachment can open that empty file, start SQLite in WAL mode, and erase the exact foreign sidecar that caused the earlier refusal. An independent disposable Linux/native SQLite 3.46.1 schedule reproduced sentinel loss for all 3 suffixes on retry; these are standalone reproductions, not Bazel/Rust passes.

Patch: both attach and attach_read_only now call OpenedDatabaseFile::pin_initialized before starting SQLite workers. It verifies the completed SQLite 16-byte file magic through the already pinned descriptor, avoiding an extra Unix close that would drop unrelated POSIX locks. Missing/invalid headers fail closed, rather than allowing SQLite to bootstrap an incomplete basename. Added falsifiable Rust regression for each occupied suffix's failed-create → existing-retry sequence, plus invalid-header refusal and valid initialized DB acceptance.

Independent native SQLite external read-only subprocess / concurrent writer probes also passed 32 nonzero Linux assertions, 8 each in rollback and WAL modes (reader result and quick_check). Neither those probes nor this review count as repository Bazel, hosted CI, native macOS or Windows qualification. New-head hosted CI must run nonzero //crates/tracedecay-rusqlite-runtime:unit_test and all relevant integration gates; macOS still needs a native workflow-dispatched run.

Not complete acceptance: an initialization that got far enough to write a valid database header can still leave an incomplete family, and post-admission POSIX pathname replacement lacks a stronger private ownership/VFS fence. Daemon-level external sqlite3 endurance, restart recovery and platform verification remain open. Keep #4513 OPEN and this PR on review/verification hold. Ordinary merging belongs to the existing PR-maintenance lane; do not merge held draft #4776 (Kev/decision model).

Refs #4513

Remove the stale discard_created call left after unsafe pathname cleanup
was deleted. Abort now preserves the database and all sidecars, releases
shared descriptors only after normal worker teardown, and transfers a
lost-file initialization pin to the existing deferred worker cleanup.
Reject initialization publication once drain or close has begun.

Add real runtime regressions for abort/read-back/stale handles, separate-
process replacement of the database and WAL/SHM/journal after worker
close, idempotent abort, and a lost writer held across initialization abort.
These Rust tests still require nonzero Bazel and native macOS execution;
no Cargo or direct Rust invocation was used.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refs #4513 — code review of exact head 1738c8e1de983e062053fb2544c62f3a9c222300.

Three actionable safety findings are attached inline: (1) invalid-header rejection closes an independent raw database descriptor and can release another local SQLite connection's POSIX locks; (2) the SQLite magic does not establish completed initialization or authorize reusing a preserved sidecar family; (3) sidecar reservations do not fence a replacement between reservation and SQLite's first access.

I reproduced the mechanisms in disposable Linux fixtures with native SQLite 3.46.1: the rejected-header/raw-close schedule changed an independent subprocess writer from SQLITE_BUSY to admitted before the original transaction ended; a separate-process post-reservation replacement lost sentinel bytes for each of WAL/SHM/journal; and a header-bearing but otherwise uninitialized WAL database accepted a retry that overwrote/deleted each sentinel. These are standalone mechanism reproductions, NOT execution of this repository's Rust/Bazel tests, and not macOS/Windows qualification.

Keep the removal of unsafe pathname-based cleanup. The next change needs production-path regressions through attachment startup/failure/retry, rather than only calling repository_start_failure directly. Require nonzero Bazel results on the resulting exact head, plus native macOS locking/sidecar coverage and relevant Windows coverage. This review is not an approval or a claim of master completion. Keep #4513 open; ordinary merging remains with the PR-maintenance owner. Do not merge or modify held draft #4776 or release #4763.

.map_err(|_| OpenedDatabaseFileError::Reserve)?;
let journal = reserve_new_sidecar(&sidecar_path(path, "-journal"))
.map_err(|_| OpenedDatabaseFileError::Reserve)?;
opened.created_sidecars = Some(Arc::new(Mutex::new(Some([wal, shm, journal]))));

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Retained reservation descriptors do not protect the sidecar directory entries used by SQLite.

After these O_EXCL creations succeed, another process can atomically replace just sessions.db-wal (or -shm/-journal) while leaving the main database unchanged. The retained handle still pins the old sidecar inode, but the worker opens the replacement by pathname; main-file identity/HAS_MOVED checks do not detect that sidecar-only substitution.

A disposable Linux/SQLite 3.46.1 probe reproduced foreign sentinel loss for all three suffixes after a separate subprocess performed exactly that replacement and before the first WAL initialization. Holding the reservation fds throughout did not prevent it.

The new cross-process test swaps files and calls repository_start_failure directly, so it proves the removal of explicit unlink cleanup, not that SQLite startup preserves replacements. Add the sidecar-only replacement schedule through actual attachment startup, including failure/retry. Establish an ownership boundary that remains valid during SQLite's sidecar operations; another pre-open stat/recheck or an advisory lock ignored by the replacing process is not a race-safe fix.

Apply the exact rustfmt changes reported by hosted Bazel Repository
gates without touching SQLite behavior or other PR files.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — repaired the exact four formatting differences reported by the failed Repository gates job at the peer's head a3c0a165. Commit 2e9b650 is a nonforce expected-old-head fast-forward and changes only formatting in the three pre-existing SQLite files, preserving the peer's new abort-initialization tests and production changes. New exact-head CI is queued; no Bazel Rust pass is claimed yet. This does not resolve the three existing P1 review findings: rejected pin/header read lock lifetime, valid-header incomplete-initialization retry, and sidecar-only replacement after O_EXCL reservation. Native macOS/Windows tests and daemon/external sqlite3 endurance also remain outstanding. Keep #4513 open and this PR on review hold. Do not merge protected draft #4776 or touch release #4763.

Avoid opening a raw database descriptor when an existing attachment
cannot contain the SQLite header. Closing that descriptor could unlock
another live SQLite writer; cover the refusal with a cross-process test.

This does not fix header-bearing incomplete stores or sidecar swaps.
Refs #4513
SQLite's minimum initialized database occupies one 512-byte page.
Refuse header-bearing shorter files without opening a raw Unix handle;
otherwise they bypass magic-only admission and can release live locks.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — published narrow POSIX-lock preservation follow-ups on this PR: e6ea022 and 83d0221, via exact-old-head, non-force fast-forwards, preserving all prior commits.

pin_initialized previously opened a fresh raw Unix descriptor even when an existing database was too short to contain an initialized SQLite page; its error return closed that descriptor and could release another live SQLite connection's POSIX writer locks. Unix now refuses files shorter than SQLite's minimum 512-byte page by metadata alone, without opening/closing a raw descriptor. Added a real Rust subprocess lock regression for the zero-byte BEGIN IMMEDIATE case (blocked before and after refused admission, allowed after rollback), plus short-file cases bearing the correct 16-byte magic (lengths 16, 100, 511), which the original magic-only check incorrectly admitted. SQLite file-format minimum: https://www.sqlite.org/fileformat2.html.

Independent Linux system SQLite 3.46.1 probe confirmed the pre-fix raw-close lock loss and 5 nonzero metadata-only mitigation assertions; these are not repository Rust/Bazel passes. Hosted exact-head Bazel //crates/tracedecay-rusqlite-runtime:unit_test, broader applicable CI, and native macOS/Windows qualification remain unverified.

Scope is deliberately partial: this does not make invalid-header refusal safe for files >=512 bytes; a raw pin still closes on those errors. It also does not prove a header-bearing store completed initialization or protect the sidecar names against noncooperating POSIX replacement after reservation. Keep the earlier three P1 review threads open and PR on review hold. Require an ownership-safe lifecycle and daemon+external sqlite3 endurance/restart tests before #4513 can be closed on master. No merge/release/draft #4776 changes.

Existing-mode admission must not open sidecars for databases whose
file size cannot be a whole number of SQLite pages. Refuse those paths
using metadata before opening a descriptor, preserving foreign journal
entries and avoiding POSIX lock-loss on the refusal path.

Add production attachment regressions for truncated header-bearing
files and WAL, SHM, and journal sentinel preservation.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 761cc49 to this issue-only PR, fast-forwarding exact old head 83d0221 without force and preserving concurrent changes.

Additional defensive admission fix: an existing SQLite DB must consist of complete pages; every legal SQLite page size is a multiple of 512 bytes. The prior >=512-byte check admitted header-bearing files of size 513 or 1025. Those could still reach the SQLite worker and allow it to delete or change unrelated journal/SHM sidecars despite the database being incomplete. Admission now rejects non-512-aligned lengths using filesystem metadata before opening a raw descriptor (avoiding an additional POSIX close on this refusal path), on all supported platforms.

Rust regressions added: direct existing-header tests for 513/1023/1025 bytes, plus a test through real RepositoryPhysicalAttachmentFactory::attach asserting InvalidHeader and byte-exact preservation of sessions.db and foreign -wal/-shm/-journal sentinels for 513/1025-byte databases. No CI bypass or test weakening.

Independent Linux SQLite 3.46.1 mechanism probe (NOT Bazel/Rust tests): of 6 unprotected invalid-page-length cases (2 lengths x 3 suffixes), 2 journal sentinels were deleted, and 6 metadata-only refusal cases preserved all original sentinels. The probe did not demonstrate every suffix is destroyed in this specific invalid-header schedule; it did confirm a real journal-loss path.

Still outstanding: this is only a necessary structural guard, not proof of completed initialization or sidecar ownership. Page-aligned but invalid or partially initialized DBs, a header-read failure after a fresh raw pin, and post-reservation pathname replacement remain unresolved. Native macOS/Windows, full external sqlite3 daemon-reader endurance, and nonzero exact-head Bazel tests are not yet proven. CI has not yet reported a run for this commit. Keep this PR under review hold and issue #4513 OPEN; no master completion/merge is claimed. Mixed draft #4776 and release #4763 remain untouched.

Match the pinned Bazel rustfmt_check_2024 output for the refused
empty-attachment writer-lock regression without changing its behavior.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — fixed the exact-head hosted Bazel Repository gates rustfmt failure on #4805 without changing behavior. Commit 9a6e93d changes one line-wrap in crates/tracedecay-rusqlite-runtime/src/connection/path_identity_tests.rs, matching the pinned rustfmt diff at line 177 from failed run 42452325096 (job 128643991644). This was committed as a child of verified old head 761cc49 and pushed with force=false and expected_sha lease; all peer work remains intact. New-head CI is queued, not a Bazel passing result.

Independent disposable Linux SQLite 3.46.1 checks this run reproduced the successful-init overwrite of pre-existing -wal/-shm/-journal bytes (3/3), demonstrated 3/3 exclusive-create refusals preserving sentinels, and verified separate-process WAL read of committed row 1 while an uncommitted row 2 existed, plus external writer exclusion/release in both WAL and DELETE modes. These probes do not run this Rust code, prove the new-head Bazel tests, or qualify native macOS/Windows.

The existing remaining holds are unchanged: full-page malformed/retry lock safety, complete initialization/sidecar ownership, post-reservation substitution, ordinary external sqlite3 + isolated-daemon endurance/restart, native macOS/Windows and proof on master. Keep #4513 open and PR-merging with the existing maintenance lane. Never merge held draft #4776 or touch release #4763.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 45d0ed2 as an issue-focused regression-only update to this existing PR. The test now exercises RepositoryPhysicalAttachmentFactory::initialize, attach, and attach_read_only for each pre-existing -wal, -shm, -journal sentinel, verifying Reserve refusal, invalid-header retry refusal, unchanged zero-byte main file and byte-exact preservation of foreign sidecars. It covers the real factory entry points rather than only calling the pin helper. The commit is a one-file, 61-line addition and a verified nonforce expected-old-head fast-forward from 9a6e93d, preserving peer commits. Exact-head CI is pending; no repository Bazel or native macOS/Windows run is claimed. This does NOT fix post-reservation replacement, page-aligned incomplete initialization or full external-sqlite3 daemon endurance. Keep #4513 OPEN; ordinary merges remain with the maintenance owner. Draft #4776 and release #4763 remain untouched.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Owner Author

Refs #4513 — issue-specific follow-up published on #4805 (non-force, expected-old-head fast-forwards retaining prior and peer commits): b154e6b replaces the existing-store raw-file header read with a read-only immutable SQLite-VFS admission probe before acquiring the independent raw pin, and 02783ec adds a falsifiable Unix cross-process POSIX RESERVED-byte regression. 935d3d4 only normalizes test imports. Four SQLite-only files changed across these commits.

Specific fix: previously a page-aligned malformed database with a bad SQLite header could be opened via File::open for inspection, then refused and closed. On Unix, that extra close can release another live connection's process-owned record locks. The immutable SQLite probe is read-only, ignores WAL/SHM/journal and uses the SQLite VFS for its open/close lifecycle. The factory and direct tests now reject page-aligned false-header stores (512/4096 bytes) without altering foreign sidecars. A separate test preserves acceptance of a legitimate live WAL store whose schema changes remain uncheckpointed. The new subprocess test inspects SQLite's actual RESERVED advisory-lock byte before and after corrupt-header rejection, with a preheld writer and without vacuous zero-test acceptance.

Independent disposable Linux SQLite 3.46.1 mechanism checks passed: 6 malformed aligned-header/sidecar preservations and 10 external writer-lock/immutable-probe assertions, plus a separate direct POSIX-fcntl schedule demonstrating exclusion survives SQLite immutable refusal whereas an independent raw open/close releases it. These checks are NOT the Rust regressions, repository Bazel, or native macOS results.

Validation hold: exact-head CI run 42465671210 was queued when checked. Require nonzero bazel test //crates/tracedecay-rusqlite-runtime:unit_test --test_output=errors on this SHA, plus affected integration/format gates and a native macOS workflow run on the exact head. The full underlying issue is not solved: the immutable main-file probe does not prove completion of initialization or own the sidecars; post-reservation POSIX sidecar replacement, safe recovery from partial initializations, ordinary daemon + external sqlite3 endurance/restart, and native platform evidence remain open. Keep #4513 OPEN; do not merge from this comment. Ordinary PR merging stays with the maintenance lane. Held draft #4776 and release #4763 remain untouched.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published issue-specific formatting repair fcd316b6ca08 directly on this PR using an exact-head, nonforce fast-forward from 935d3d4d8733; the remote is now at fcd316b6ca08, one commit ahead / zero behind that prior head. Only connection/mod.rs and connection/path_identity_tests.rs changed, with the three formatting hunks requested by Bazel's pinned rustfmt.

The preceding exact-head CI run 42465671210 failed Repository gates at //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024 on precisely those hunks (immutable-inspection let wrapping and two OpenOptions chain layouts). Other CI jobs were still pending/queued when checked, not passed. This commit is behavior-preserving and does not fix the known post-reservation replacement race or incomplete-initialization recovery. Do not infer Rust unit-test or macOS acceptance until actual nonzero results on this new SHA.

Keep #4513 open; ordinary PR merging is owned by the existing maintenance lane. Held draft #4776 and release #4763 are untouched.

Refs #4513. The prospective SQLite initializer now owns empty
-journal reservations as well as WAL/SHM. Require the scoped reset
tests to account for those exact owned files and retire the journal
in the shipped-store fixture before attempting a new installation.

The hosted Linux fleet at b6c7cd3 reproduced the old assertions and
leftover journal in the test fixture; these assertions remain strict.

Copy link
Copy Markdown
Owner Author

Refs #4513 — current-head Linux Bazel regression triage and narrow test repair.

The exact PR head b6c7cd3bf2d61a7ca51a7343d6c167c642bd7278 Linux fleet job failed four test targets, not a CI infrastructure/disk-capacity failure. Inspection of the hosted Bazel logs found:

  • //crates/tracedecay:transport_acceptance_suite (28 pass / 2 fail): scoped reset correctly removes the global.db-journal and sessions.db-journal reservations as well as main/WAL/SHM; the exact-deletion assertions still expected the old three-member family.
  • //crates/tracedecay:transcript_ingest_suite (205 pass / 1 fail): the shipped-store reset fixture deleted only main/WAL/SHM, leaving an occupied -journal reservation before it attempted the fresh prospective installation.
  • //crates/tracedecay-global-db:unit_test (497 pass / 3 fail): three tests pass an existing, schema-unpublished SQLite main (empty schema / only writer ledger) to attachment; the tightened pin_initialized properly refuses ambiguous sidecar ownership. This needs a test-fixture/product-contract resolution via a genuinely published prospective store, not weakening the ownership admission gate.
  • //crates/tracedecay:runtime_acceptance_suite (140 pass / 1 fail): real daemon startup can fail finish prospective SQLite schema: SQLite runtime is busy; the finish command currently uses a nonblocking try_send and may encounter a full writer command channel. This requires queue/deadline diagnostics and regression-backed resolution, not a blind timeout increase.

Published focused commit b3c9629564ce to #4805 with the three test-only changes: exact reset inventories now require journal removal, and the shipped-store reset fixture retires the journal as well. Branch update was expected-old-head (b6c7cd3), nonforce fast-forward; verified exactly one commit ahead and zero behind, affecting only those three tests. These changes do not weaken reset assertions or alter production cleanup. New exact-head CI is queued; no green Bazel result is claimed for the new head.

Standalone Linux SQLite 3.46.1/Python subprocess probes passed 40 nonzero checks for WAL reader/writer exclusion and committed-data visibility, integrity/restart, occupied journal refusal and four-member scoped removal. These are not repository Rust/Bazel tests or native macOS qualification.

The existing cross-process canonical namespace/sidecar race under a noncooperating same-UID writer is still an independent acceptance blocker. master is 8d6fe6791a1b9983a066f9ada990d70cb6eb451a; #4513 stays open. Separate #4803 is merged. No merges were attempted. Protected draft #4776, release #4763, and the independent PR-maintenance workflow remain untouched.

Refs #4513. Treat prospective schema finalization as a serialized writer
barrier rather than a best-effort try_send. Deterministic bounded-channel
regressions cover backpressure, FIFO admission, completion replies, and
closed-writer refusal without relaxing ordinary SQL admission.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 185181fb2b78 on this issue-specific PR.

Fix: ExactSqlHandle::finish_initial_schema previously used try_send against the bounded exact-SQL writer queue. The hosted Linux Bazel failure includes a real daemon acceptance failure finish prospective SQLite schema: SQLite runtime is busy. A momentarily full command queue returned Busy at the durable prospective-schema publication barrier, potentially stranding initialization. The barrier now awaits actor-channel capacity using a native Cx, retains serialization and the writer's completion reply, and still refuses a disconnected writer; ordinary SQL admission and queue capacity remain unchanged.

Regression added: two deterministic, nonzero-assertion Rust tests in exact_sql/tests/async_dispatch.rs fill a one-slot actor command queue, verify finalization remains pending rather than spuriously failing, then release capacity and require its own completion reply; a separate case checks typed refusal when the receiver closes during backpressure. The tests are checked in but have not yet been executed as Bazel results. The change is confined to exact_sql/mod.rs and exact_sql/tests/async_dispatch.rs (one one-parent commit, no other source or test changes). It was pushed from exact previous head b3c9629564ce1aa81e0a96cbd7c76def57a35785 via expected-old-head nonforce fast-forward; remote compare is one ahead, zero behind.

Exact-head hosted CI is queued, not green. No local Bazel runner is available, and no Rust test pass is claimed for this new head. The three global-db schema-unpublished fixture failures, canonical SQLite pathname/sidecar ownership race, and final native macOS/Windows and daemon/external sqlite3 qualification still require resolution/evidence. master remains 8d6fe6791a1b9983a066f9ada990d70cb6eb451a, and issue #4513 remains OPEN. No issue closure or merge was performed. Protected draft #4776, release #4763, and the separate PR-maintenance workflow remain untouched.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 45358967af30 to the existing issue-specific PR #4805, solely in two tracedecay-global-db test files.

The previous hosted Linux Bazel run reported 497/500 global-db tests passing, with three failures where test fixtures manufactured an existing schema-unpublished SQLite namespace then expected registered runtime admission to silently claim it. Production's strengthened physical attachment correctly refuses that ambiguity. This was an obsolete fixture/contract, not grounds to weaken the sidecar ownership guard.

The patch:

  • Requires safe refusal without installing a marker when an old raw SQLite file has had its last application table dropped, then verifies final schema publication on a genuinely new prospective family and restart.
  • Requires an unmarked writer-ledger-only database to remain unmodified and refused; verifies that a freshly owned prospective database still publishes the writer ledger and complete schema.
  • Makes the daemon cancellation fixture enter the real prospective initialization pathway instead of precreating a raw SQLite file. It sweeps cancellation points after publication, requiring typed cancellation, unchanged installed schema and successful re-admission. Adds a direct unclaimed -wal sentinel preservation regression using the actual registered runtime publication API. Interrupted-install and cross-process recovery regressions remain in the runtime-core suites.

The commit was a one-parent expected-old-head, nonforce fast-forward from 185181fb2b78; remote comparison confirms one ahead / zero behind and exactly the above two test files modified. master remained 8d6fe6791a1b.

Verification status: the current exact-head hosted CI is pending, so these Rust changes are not yet Bazel-verified. Independent Linux SQLite 3.46.1/Python child-process probes passed eight nonzero assertions for live WAL reader visibility, uncommitted isolation, restart integrity and unchanged unclaimed sidecar bytes; those are not repository Bazel or native macOS tests.

No issue closure or merge. The known canonical SQLite pathname/sidecar namespace race under a noncooperating same-UID process remains an independent acceptance blocker; full native macOS, Windows, daemon/external-system-sqlite3 acceptance and proof on master still govern #4513. Protected draft #4776 and release #4763 remain excluded; ordinary merging stays with separate PR maintenance.

Copy link
Copy Markdown
Owner Author

Refs #4513 — new exact-head hosted Bazel finalization blocker on #4805 45358967af30f76b252ed834435fd109d502bbf7. This is not a verified source fix or native macOS pass.

Linux fleet job 129256248869 failed 2 of 183 Bazel test targets (181 passed):

  1. //crates/tracedecay:unit_test — 795 passed, 1 failed: mcp::server::host_admission_tests::owned_project_replay_worker_backoffs_on_retryable_failure. Its real registered session-runtime initialization, before the test can exercise host replay, failed at writer_test_support.rs:106: PhysicalRuntimeFailed { operation: "finish prospective SQLite schema", message: "SQLite runtime is busy" }. The exact-head patch from 185181fb already awaits bounded writer-queue capacity; treating this as merely the previous try_send-saturation failure would be incorrect. The current WriterCommand::FinishInitialSchema in crates/tracedecay-rusqlite-runtime/src/exact_sql/command.rs also returns ExactSqlError::Busy whenever PRAGMA wal_checkpoint(TRUNCATE) does not report (0,0,0). The initialization route still starts a reader pool before schema installation and does not drain its workers until after finalization. The concrete offending connection/lease and checkpoint result are not yet measured, so the checkpoint is a source-supported hypothesis, not an asserted root cause.
  2. //crates/tracedecay-global-db:unit_test — timed out at its existing 300-second Bazel limit after emitting 500 successful test results of 501 tests. There is no proven assertion failure in this target, and the one unfinished test's identity is not established by its parallel log. Do not paper over it by increasing timeouts.

Independent SQLite mechanism proof, not TraceDecay Bazel: with SQLite 3.46.1 on Linux, a separate process held an active WAL read transaction while the writer committed a new row. wal_checkpoint(TRUNCATE) returned (1,4,3), then (0,0,0) when the reader released its snapshot. The old reader saw only the committed pre-snapshot row, and both committed rows plus integrity_check=ok survived. 10 nonzero assertions passed. The mechanism shows why a finalization barrier must account for readers; it does not establish that this specific TraceDecay failure had that cause.

Required repair/evidence: distinguish writer-channel admission from SQLite checkpoint-busy results (record the checkpoint result and actual live reader/lease blockers in repository-native diagnostics), arrange a safe publication barrier that waits for/ends the owning initialization readers without violating worker ordering or WAL durability, and add a deterministic real registered-runtime Rust regression with an overlapping SQLite reader and finalization/recovery, including WAL/SHM/journal preservation and retry. Verify using Bazel, then native macOS and Windows; keep test semantics and existing timeouts intact unless a measured irreducible budget justifies a separate change. The separately acknowledged canonical pathname/sidecar ownership race is still an independent blocker.

master advanced to 0497fd83086872ca6edfa7cad01f5a300414d255 in an unrelated two-file sessions-host change; this PR remains unmerged at the exact head above. #4513 remains OPEN. No source changes or merges in this investigation; held Kev/decision-model draft #4776, release #4763 and the independent PR-maintenance workflow are untouched.

The schema writer must retain its original reservations while all staging
reader workers exit. Retire both reader lanes before the TRUNCATE barrier,
so live snapshots cannot turn a complete schema into a false BUSY refusal.

Add a real repository attachment regression for a pinned WAL snapshot,
reader admission fencing, retained writer, and committed SQLite integrity.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — published the initialization reader-drain repair at ebbf9828.

Exact-head failure addressed: Linux Bazel fleet job 129256248869, with its archived per-target test logs, reports mcp::server::host_admission_tests::owned_project_replay_worker_backoffs_on_retryable_failure failing while opening a prospective registered sessions store: finish prospective SQLite schema: SQLite runtime is busy (795 passed, 1 failed in //crates/tracedecay:unit_test). The same fleet had a separate //crates/tracedecay-global-db:unit_test timeout after successful individual tests. Those targets are not green.

Repair: Prospective schema installation now fences general and reserved-health reader admission, waits for active/limbo snapshots to settle, and joins the staging reader workers on the blocking pool before asking the retained original SQLite writer to stamp and wal_checkpoint(TRUNCATE). Original main/WAL/SHM/journal reservations remain under the same attachment until writer shutdown and publication. A refusal leaves the draining reader pool attached for ordinary joined failure recovery; no ambiguous pathname is unlinked. Existing nonprospective schema installation remains unchanged.

Regression: Added a real Rust repository-attachment test that pins a SQLite read snapshot, writes a second committed WAL row, confirms reader finalization waits for the snapshot, refuses reopened reads after the fence, preserves the writer until finalization, and checks both committed rows plus PRAGMA integrity_check after shutdown.

Verification boundary: Eight nonzero standalone Linux SQLite 3.46.1 assertions passed against an independent-process wal_checkpoint(TRUNCATE): (1,4,3) with a live WAL reader, (0,0,0) after release, followed by committed-row and integrity verification. These are independent SQLite probes, not new TraceDecay Bazel results. Exact-head CI 42667888456 is queued; the new Rust regression, native macOS qualification, and master acceptance remain unproven. The canonical namespace/sidecar ownership race remains a separate unresolved blocker. No merge requested; ordinary non-model merging remains in the separate maintenance workflow. Protected draft #4776 and release #4763 untouched.

Match Bazel's Rust 2024 rustfmt gate for the reader-drain repair.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 77e765888ad as an issue-focused Rust 2024 formatting repair on this existing PR.

The previous exact-head hosted run failed the Bazel repository gate at //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024. The archived job 129336521870 gives five concrete rustfmt hunks in src/repository/attachment.rs introduced by the reader-drain fix. This commit applies exactly those five mechanical formatting changes, without touching safety logic, test assertions, CI, or any unrelated file. The downstream "Feature gates" and "Test Linux" summary jobs reported failure/abandoned after the upstream gate failed; no successful new Rust/Bazel test is claimed.

The GitHub compare confirms one-parent nonforce, expected-old-head update (ebbf98285ce4 → 77e765888ad) with one changed file, 1 ahead / 0 behind. The new exact-head CI is pending; prior run's Clippy was still in progress when checked. No native macOS/Windows qualification is claimed for this new head.

Independent local Linux SQLite 3.46.1 probes passed 25 nonzero assertions with an external read-only subprocess, live WAL writer, committed-row visibility, restart integrity, and O_EXCL refusal/preservation of occupied -wal/-shm/-journal names. These are standalone mechanism checks, not TraceDecay Rust/Bazel or native macOS tests.

master remains at d6acbc49f79187673dfd1ba037074ca094726d87; PR #4805 is unmerged, #4513 remains open, and the noncooperating same-UID canonical namespace/sidecar ownership race remains an independent acceptance boundary. Protected draft #4776 and release #4763 were not touched. Ordinary merging stays with the established maintenance workflow.

ScriptedAlchemy commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner Author

Refs #4513 — new exact-head hosted Linux Bazel CI blockers, with source attribution still unproven.

At PR head 77e765888ad1bd1f0f9ba4745d24e9ebb1097de7, CI run 42677456386 / Linux fleet job 129368492456 completed on 2026-11-09 with 181 of 183 test targets passing and two failures:

  1. //crates/tracedecay-sessions:unit_test: 670 passed / 1 failed. runtime::hosts::codex::tests::recent_first_discovery_tests::unsettled_bucket_native_change_invalidates_even_with_same_entry_names panicked at crates/tracedecay-sessions/src/runtime/hosts/codex/tests.rs:2723, this regression requires the unsettled-clock branch. This is a Codex timestamp-fixture/precondition failure, not demonstrated SQLite corruption; verify its native clock assumption under the fleet filesystem, independently of storage: opening a live sessions.db with the sqlite3 CLI makes the daemon report disk I/O errors / corrupt the file #4513 changes.
  2. //crates/tracedecay:dashboard_api_test: 88 passed / 1 failed. projects::project_scoped_gateway_reports_registry_read_failures_as_unavailable panicked in wait_for_dashboard at tests/common/mod.rs:1587 after readiness never reached /api/capabilities. The dashboard process did print a listening URL, so the present evidence is a failed authenticated HTTP readiness probe, not proof of an SQLite schema/initialization error. Inspect actual capability response/status, server exit state, and resource contention before touching time budgets or SQL code.

Repository gates, Clippy, feature gates, Dashboard, benchmark self-tests all passed for this exact head. These two failures have not been independently reproduced in isolation and neither is presently attributed to the SQLite patch; avoid speculative source edits or increased timeouts. The Linux summary job later completed failure solely because the Linux fleet failed (it did not run a second test suite); Windows is still independently running, and macOS was skipped. Archived Bazel logs were uploaded.

Independent local Linux SQLite 3.46.1 subprocess/filesystem reproduction passed 30 nonzero assertions: read-only readers observed only committed rows across three live-WAL write transactions, reopening passed integrity, all three occupied sidecar reservations refused O_EXCL without altering foreign bytes, and a separate-process sidecar-only substitution was subsequently overwritten by native SQLite for each of WAL/SHM/journal despite a retained original reservation descriptor. That last result reconfirms the existing open canonical sidecar P1; it is not a new fix or a repository Bazel/native macOS test. No source or PR ref was modified.

The untrusted same-UID canonical sidecar namespace still lacks a proved lifetime ownership boundary, so #4513 remains open, and even green CI would not close acceptance. master remained d6acbc49f79187673dfd1ba037074ca094726d87. No merge or close was attempted; held Kev/decision-model draft #4776, release #4763, and ordinary PR-maintenance ownership are unchanged.

Copy link
Copy Markdown
Owner Author

Refs #4513 — new POSIX WAL/SHM lock-loss exposure to qualify in PR #4805 (reviewed exact head 77e765888ad1bd1f0f9ba4745d24e9ebb1097de7). This is a native-SQLite mechanism reproduction, not a claim that a TraceDecay daemon path has yet hit the overlap, or that the repository Rust/Bazel or native macOS tests passed.

The new-store reserve_new_sidecar creates/retains ordinary O_RDWR | O_CREAT | O_EXCL descriptors for all three sidecars in created_sidecars. They are closed by release_after_workers_exit or transferred through take_completed_files. The earlier physical-main-file POSIX-lock fixes are insufficient to establish safety for this different inode: WAL's writer lock is a process-owned POSIX fcntl lock on -shm, so closing any unrelated raw descriptor for that SHM inode can release an independent same-process SQLite connection's WAL locks. This attachment's own writer/reader shutdown is not by itself proof that every independent same-process connection using the SHM inode has finished.

Six isolated native Linux SQLite 3.46.1 trials, 36 nonzero assertions, actual separate-process writer: O_EXCL-reserve sessions.db, -wal, -shm, -journal using the same ordinary descriptor types as the implementation. Start SQLite WAL, commit value 7, verify SQLite reused the exact reserved SHM inode, then hold BEGIN IMMEDIATE and an uncommitted write. A separate compiled C/libsqlite3 process's BEGIN IMMEDIATE returns SQLITE_BUSY (5) before closing the original reserved -shm fd. Closing only that raw -shm fd admits the external process's writer (SQLITE_OK (0)) while the original transaction remains open, in all 6/6 trials. Roll back the original transaction: the previously committed 7 survives and PRAGMA integrity_check returns ok (no corruption claimed from this probe). Independent control runs showed closing a raw -wal or -journal fd did not admit the writer; neither did closing an O_PATH pin to the -shm inode. All five control scenarios passed 30 further nonzero assertions. This is a different mechanism from closing a raw main-DB pin.

Actionable acceptance work: audit whether an independent same-process SQLite handle can overlap created_sidecars release or interrupted-initialization transfer (including error paths), then give the SHM reservation a truly SQLite-lock-neutral or VFS-coordinated ownership/close lifetime, not just “after this attachment's workers exit.” Cover the actual OpenedDatabaseFile::create_new/release and recovery production paths with a deterministic Rust subprocess-writer test and native macOS qualification; Linux O_PATH is a useful control but not a portable macOS solution. Keep the foreign WAL/SHM/journal preservation/namespace race requirements intact.

No source commit, CI/test relaxation, merge, or issue closure was made in this investigation. The previously known canonical basename replacement race remains unresolved; #4513 stays open.

Transfer creation pins into publication and close all raw main/sidecar
verification handles before clearing the admission receipt. Apply the
same ordering to prepared publication and published-family recovery.

Add deterministic Unix subprocess regressions that admit a real SQLite
connection at the completed-handoff event and verify rollback/WAL read
locks, independent WAL writer exclusion, release, and restart integrity.
Preserve refusal when cold recovery lacks durable allocation generations.

Standalone Linux SQLite 3.46.1 mechanism checks passed; repository Bazel
and native macOS qualification remain required on this exact commit.

Refs #4513

Copy link
Copy Markdown
Owner Author

Refs #4513 — published a production handoff repair and four Unix regression tests in bc5cbc1fc5ad78e05748d1e73628f0e92296e74f.

Repair

The raw-SHM report led to a concrete ordering defect in the publisher: retire() cleared the preparation receipt while publish_prepared's verification descriptors, the outer publish creation descriptors, or recover_published's descriptors were still in scope. verify_read_only_handoff observes the receipt without acquiring the writer lease, so an admitted connection could acquire SQLite locks before a remaining raw main/SHM descriptor closed.

Fresh publication now transfers creation handles into the same owned collection as the verification handles. That entire collection is closed before the receipt is cleared. Prepared publication and already-published recovery apply the same order. Identity/content checks, no-replace publication, foreign-file preservation, durable receipt handling, and refusal on unproven allocation generations are unchanged. The completed-handoff tracing event now records the actual released boundary.

Regression coverage

The new initialization::tests::handoff module uses the canonical tracing-capture helper to admit a real SQLite connection exactly at that production boundary, without sleeps or test-only runtime hooks. A separate libtest subprocess must remain excluded after publisher teardown. It covers rollback read snapshots, WAL read snapshots, independent WAL writer exclusion, ordinary acquisition after release, committed data and restart integrity across fresh/prepared publication and published recovery. Cold recovery on filesystems without allocation generations explicitly exercises preservation/refusal; it does not pretend to qualify successful recovery there. The child process has a completion marker to reject a zero-test subprocess green.

Evidence and validation hold

  • 8 standalone Linux/native SQLite 3.46.1 tests passed. Negative controls reproduced lost rollback-reader/WAL-reader/WAL-writer exclusion when raw descriptors close after admission; the release-before-admission controls preserved exclusion and restart integrity. Additional cases preserved occupied WAL/SHM/journal names and foreign cross-process replacements. These are Python/native SQLite mechanism tests, not execution of the Rust regressions.
  • The original source was reconstructed byte-exactly and checked against blob 9933b1d56d586badfbfbaec1c46af82b889344f2 before editing; both published blobs match the locally reviewed bytes. Compare confirms exactly two files, one commit ahead/zero behind the former PR head. Publication used force=false and expected old head 77e765888ad1bd1f0f9ba4745d24e9ebb1097de7.
  • Exact-head CI is pending, not passed. No local Bazel/Rust execution is claimed; Cargo was not used.

Required exact-head checks:

bazel test //crates/tracedecay-runtime-core:unit_test --test_output=all --test_arg=initialization::tests::handoff
bazel test //crates/tracedecay-runtime-core:unit_test //crates/tracedecay-rusqlite-runtime:unit_test --test_output=errors
bazel build //crates/tracedecay-runtime-core:rustfmt_check_2024

Confirm nonzero execution of the four behavioral regressions, and run the same affected targets natively on macOS using the existing workflow. The available connector exposes job reads/reruns but no workflow-dispatch action; no workflow files or infrastructure were altered.

This repairs the publication/pin-lifetime window, not arbitrary same-user mutation of the active canonical SQLite namespace. That separate P1 limitation, current-head native qualification, daemon acceptance, and verification on master remain open. No approval, merge, or issue closure; ordinary merges remain with PR maintenance. #4776/model work and release #4763 are untouched.

Refs #4513. Apply the exact three rustfmt 2024 hunks reported by
hosted Bazel repository gates for bc5cbc1; no behavioral edits.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published a narrow hosted Bazel Rust 2024 formatting repair on the existing issue-specific PR #4805: 714474f9dc8e.

The previous exact-head CI run 42703851238 failed //crates/tracedecay-runtime-core:rustfmt_check_2024, not the Linux Rust suite. Its log specified exactly three hunks in src/shard_runtime/registry/initialization/tests/handoff.rs; this commit applies those three exact formatting changes with no behavioral, fixture, or assertion changes. The other repository-gate tests passed (8/8); Clippy, Dashboard, and the hosted Linux jobs succeeded on the previous head. The dependent Feature gates job failed because Repository gates failed; macOS was skipped, and Windows qualification was cancelled on that previous run.

One-parent, issue-focused commit; the branch advanced by expected-old-head (bc5cbc1fc5ad) nonforce fast-forward and remote re-read verified the exact published SHA and the one-file diff. Exact-head CI 42720083882 is queued, not passed.

A separate disposable Linux SQLite 3.46.1 subprocess baseline passed 57 nonzero assertions: read-only SQLite processes observed every completed WAL commit, restart/integrity checks passed, and O_EXCL correctly refused all three occupied WAL/SHM/journal names. These are standalone tests, not repository Bazel, an actual TraceDecay daemon, or native macOS results.

The existing P1 canonical sidecar namespace replacement review remains unresolved and requires a real ownership boundary, not another pathname recheck. PR #4805 is not master, and issue #4513 stays OPEN. No merge or change to draft #4776, release #4763, or the independent PR-maintenance workflow.

Refs #4513.

Use Arc::try_unwrap to fence late WeakReaderPool upgrades before
acknowledging initialization-reader or attachment shutdown. Restore the
same pool and retry within the existing deadline when another owner won.

Track worker owners extracted for out-of-lock joins so a concurrent lost
file callback cannot report closure and release main/sidecar pins while
SQLite connections are still alive. Preserve a failed join on retries.

Add deterministic production-path races and independent SQLite subprocess
checks for writer exclusion, committed data and integrity. Rust/Bazel and
native macOS execution remain pending; local standalone Linux SQLite
mechanism probes are not repository test results.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review repair published: shutdown ownership fences

Refs #4513. Published 89f84bd672c9 on this issue-specific branch. This is not an approval or a claim of default-branch completion.

Source defects repaired

  1. A quiescent Arc count did not establish exclusive shutdown ownership. Both initialization-reader close and ordinary attachment drain observed ReaderPool::is_quiescent, then dropped the pool as though that necessarily joined its SQLite workers. An exact-SQL weak handle can upgrade between the observation and retirement. The surviving strong reference would keep the workers alive after the caller acknowledged reader shutdown. ReaderPool::try_into_shutdown now uses Arc::try_unwrap to atomically claim the actual PoolInner and fence subsequent weak upgrades. A failed claim restores the same pool and retries within the existing deadline. Successful owners join outside the attachment mutex.

  2. Worker owners temporarily extracted for an out-of-lock join were invisible to lost-file observations. During that interval, the writer's lost-file callback can see empty normal slots. lost_file_closed could then treat empty normal/lost slots as proof of closure and release main/sidecar identity pins while SQLite connections were still live. The new workers_closing state covers that interval, prevents premature closure/pin release, and clears only after the joins return. close_and_join also requires completed drain. A writer join error is retained so a later drain cannot report success merely because its slots are empty.

The repair does not add any pathname deletion or relax existing WAL/SHM/journal preservation checks.

Regression coverage added

  • a_late_weak_upgrade_cannot_outlive_an_acknowledged_reader_shutdown
  • initialization_reader_close_waits_for_a_late_weak_upgrade
  • attachment_drain_waits_for_a_late_weak_upgrade
  • lost_notice_during_out_of_lock_join_preserves_pins_and_writer_exclusion (Unix)

The attachment regressions use real SQLite and deterministic one-shot hooks at the production ownership boundary, not a timing-only attempt to win the race. Fresh subprocesses check writer exclusion, committed-row visibility, and integrity_check. The snapshot helper requires successful exit, exactly one executed test, and a completion marker. The lost-notice test deliberately injects the real callback while the files remain in place so an independent process can probe the original locks; it does not claim to simulate a filesystem rename.

Verification and limits

Remote comparison confirms exactly one commit ahead, zero behind the inspected 714474f9dc8ed0362c7088d83cc36c143574795d, changing four SQLite runtime files. The branch was re-read immediately before an expected-old-head, nonforce fast-forward, and the resulting PR head was verified.

Fourteen nonzero standalone Linux SQLite 3.46.1 assertions passed. They demonstrate the raw-descriptor-close lock-loss mechanism for a rollback-mode main file and a WAL SHM file using independent subprocess writers, followed by committed-data and integrity checks. These are not execution of the new Rust regressions, repository Bazel results, a TraceDecay daemon journey, or native macOS qualification. Bazel is unavailable in this runtime; Cargo and direct rustc were not used.

Exact-head hosted CI for 89f84bd is queued. The new source still requires compilation, meaningful nonzero Bazel execution, and native macOS qualification. Previous-head platform results do not qualify this commit.

Keep #4513 open and the existing independent merge hold in place. This lifecycle repair does not resolve the previously documented canonical namespace/sidecar replacement P1 or prove full acceptance on master. No merge, issue closure, protected #4776 change, release #4763 change, CI weakening, or PR-maintenance automation modification was made.

Refs #4513

Apply the exact Rust 2024 rustfmt hunk from hosted repository gates
without changing shutdown ownership, SQLite behavior or test assertions.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published a narrow exact-head CI formatting repair to this existing issue-specific PR: d80c1b617abd.

The prior exact-head hosted Bazel repository-gates job failed //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024. Its complete error output identified one exact rustfmt hunk in src/repository/attachment.rs:597, joining a multiline if quiescent && let Some(readers) = state.readers.take() condition onto one line. The commit applies only that mechanical adjustment (one file, +1/-3 lines), without changing reader shutdown ordering, SQLite policy, tests, workflows, or dependencies. All eight repository-gate tests in the previous run passed; the rustfmt build failure made the overall gate red.

The GitHub write used an expected-old-head (89f84bd672c9), nonforce fast-forward; remote re-read and compare confirmed exactly one commit and one modified file (1 ahead, 0 behind). Fresh exact-head CI #42748402187 is pending, not a Bazel pass. The preceding run's Linux fleet and Clippy remained queued at last inspection; macOS was skipped.

Separately, 17 nonzero local standalone Linux SQLite 3.46.1 cross-process assertions passed for WAL writer exclusion, external read-only visibility before/during/after commit, restart integrity, and occupied WAL/SHM/journal preservation. These do not execute TraceDecay Rust/Bazel or native macOS.

No acceptance claim or merge: issue #4513 stays OPEN. The documented canonical sidecar namespace replacement P1 and native macOS/current-head regression qualification remain outstanding. master is aa03f27ef074 and the issue-specific PR is unmerged. Protected Kev draft #4776, release #4763, and independent PR-maintenance ownership were untouched.

Refs #4513.

Require exact single-entry names before either native no-replace rename
or sibling replacement. A dirfd does not confine absolute paths, parent
traversal or intermediate symlinks. Reject normalized aliases and NUL
without opening any SQLite descriptor or mutating either namespace.

Add filesystem preservation regressions for the database and all three
sidecars, plus an independent-process intermediate-directory replacement.
Keep the separate full-path framed-log API and existing positive tests.

Native Linux renameat2 reproduces all three escapes with a committed
SQLite row. Repository Bazel and native macOS qualification are pending;
this does not resolve the remaining same-UID canonical leaf race.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 73b01ef17b85, a production namespace-boundary repair plus regression tests. Updated this branch from exact old head d80c1b617abd13b2698a61f15e9d1758a79ef81d using expected_sha and force=false; remote verification confirms the new head. The comparison is exactly one new commit, zero behind its parent, and two changed files.

Defect and repair

capability_dir::rename_noreplace forwarded arbitrary source/destination paths to native rename operations. A held directory descriptor does not confine ../foreign/sessions.db, an absolute path, or bridge/sessions.db when another process replaces bridge with a symlink. Those paths can move a foreign file despite the advertised single-entry capability contract.

Both rename operands now require an exact single normal entry before any filesystem mutation. The shared validator also rejects empty/dot names, normalized aliases such as payload/., and NUL, and is reused by sibling replacement. It does not open another SQLite descriptor or introduce an inode-check/unlink race. The separate full-path framed-log API is unchanged. Current SQLite publication uses fixed or extracted leaf names; this repairs the underlying primitive's missing boundary, not a claim that a normal registered caller currently supplies traversal.

Regression coverage and verification limits

Added three behavioral Rust regressions plus a child-process helper in capability_dir/namespace_tests.rs: source/destination traversal and absolute-path refusal preserving the main/WAL/SHM/journal fixtures; malformed-name refusal for both rename variants; and an independent process replacing an intermediate directory after its parent is pinned. The subprocess test requires both a completion marker and a nonzero executed-test result. Existing successful-leaf and occupied-destination tests remain unchanged.

An independent disposable Linux/native renameat2 reproduction using real SQLite 3.46.1 databases reproduced all three escapes (parent traversal, absolute source, child-rebound intermediate symlink). In each case the foreign database's committed row was moved and remained readable/integrity-valid: 12 nonzero assertions. This is a standalone mechanism reproduction, not execution of the new Rust tests or proof of the patched repository behavior.

This runtime has no Bazel installation; Cargo was not used. Required repository verification remains bazel test //crates/tracedecay-private-fs:unit_test --test_output=errors and bazel build //crates/tracedecay-private-fs:rustfmt_check_2024, plus the relevant runtime/SQLite suites. Exact-head CI is pending. Native macOS/Windows qualification is not claimed.

Keep #4513 open and retain the merge hold. Rejecting path escapes does not make replacement of a valid canonical leaf by a noncooperating same-UID process safe. That separate namespace-ownership requirement and default-branch acceptance remain outstanding. No master merge, protected #4776 change, release #4763 change, infrastructure change, or maintenance-task modification was made.

Refs #4513.

Require HAS_MOVED attestation when verifying a live SQLite worker. An unchanged path cannot establish the identity of a VFS handle when file control is unsupported. Exercise the actual SQLite memdb VFS and confirm a normal disk worker still attaches to the unmodified store.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published 12f5b305b6b9 on this issue-specific PR.

Worker identity fail-closed repair: OpenedDatabaseFile::verify_connection previously accepted SQLITE_NOTFOUND from SQLITE_FCNTL_HAS_MOVED because its check rejected only Some(true). A stable canonical pathname authenticates the name, not an unverified SQLite VFS handle. Worker verification now refuses missing file-control proof with OpenedDatabaseFileError::Inspect; a positive Some(true) still rejects replacement and a supported Some(false) retains both pathname checks. The existing verified-reader and Darwin initial-pin paths already refused None. No sidecar deletion, extra raw SQLite descriptor, policy relaxation or CI change was introduced.

Real-VFS Rust regression added: worker_identity_refuses_a_vfs_without_move_attestation uses SQLite's built-in memdb VFS against the same on-disk pathname. It checks the actual SQLITE_NOTFOUND result, requires worker verification to refuse it, then opens the actual on-disk database through the ordinary VFS, verifies positive admission and reads the original committed row. This should fail before the repair, and tests the production method rather than a mocked file-control result.

Verification boundaries: 15 nonzero standalone Linux SQLite 3.46.1 C assertions passed: the ordinary file VFS returned SQLITE_OK for HAS_MOVED, memdb returned SQLITE_NOTFOUND, and the committed on-disk SQLite header remained intact. These are mechanism probes, not execution of the repository Rust test, Bazel, a daemon, or native macOS. This runtime has no Bazel runner; Cargo was not invoked. Require nonvacuous bazel test //crates/tracedecay-rusqlite-runtime:unit_test --test_output=errors plus the Rust 2024 formatting gate and current-head Linux/macOS qualification. Exact-head hosted CI #42764933700 was queued at last check.

The branch update used the exact previous head 73b01ef17b85, expected_sha, and force=false; post-write comparison confirms one commit (ahead 1, behind 0), exactly two SQLite-runtime files changed. Issue #4513 remains open: this closes a missing-VFS-attestation hole, not the unresolved same-UID canonical WAL/SHM/journal leaf-replacement boundary or full default-branch acceptance. master remained 2f9d659 at write time. Protected draft #4776, release #4763, and the separate PR maintenance workflow were not touched.

Refs #4513

Apply the four mechanical Rust 2024 formatting changes reported by
hosted Bazel run 42764933700. Preserve all assertions and behavior.

Copy link
Copy Markdown
Owner Author

Refs #4513 — published an exact-head Bazel formatting repair on the existing issue-specific PR.

Commit: b1f093e8966e

The previous head's hosted Repository gates completed with eight Bazel test targets passing but failed two rustfmt_check_2024 build actions. The job log names exactly four formatting hunks: three in crates/tracedecay-private-fs/src/capability_dir/namespace_tests.rs and one in crates/tracedecay-rusqlite-runtime/src/connection/tests.rs. This commit applies those four mechanical changes, without altering production code, test assertions, dependencies, policies, or CI. GitHub comparison verifies one new commit and exactly those two files; the update used expected_sha=12f5b305b6b9 with force=false and the exact new branch head was read back.

New exact-head hosted run: #42773456440 is queued, not yet passed. Independently, 18 nonzero standalone Linux SQLite 3.46.1 assertions passed across three disposable databases, with subprocess read-only WAL snapshots during a pending write, commit visibility, sidecar presence, reopen and integrity_check. Those are not repository Rust/Bazel or native macOS results. No local Bazel runner or checked-out source tree is available in this environment; no Cargo used.

Hold: PR remains unmerged, issue #4513 remains OPEN, native macOS qualification of the latest source remains outstanding, and the canonical sidecar-namespace same-UID replacement boundary is unresolved. Existing draft #4776, release #4763 and ordinary PR-maintenance ownership remain untouched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant