Repository navigation
fix(storage): stream growth measurements and preserve registry errors - #4794
ScriptedAlchemy wants to merge 67 commits into
Conversation
|
Refs #4099. Aggregate each table's dbstat payload inside a correlated subquery instead of materializing all pages for GROUP BY. Preserve the existing reader snapshot, cancellation, deadline and typed-error path. Add nonzero filesystem-backed regressions for exact payload attribution, growth, logical deletion versus physical compaction, VM-work reduction, and interruption followed by a successful retry. No retention policy or measurement budget is relaxed; full 24-hour profile acceptance remains open.
Refs #4099. Apply the exact rustfmt delta reported by the hosted Repository gates; no production logic or acceptance criteria changed.
Refs #4099. Use a value assertion so the hosted Clippy assert_is_empty policy can display an unexpected store on failure.
Refs #4099. LIKE treats the underscore in sqlite_% as a wildcard, silently excluding valid user tables such as sqlitemessages. Match the literal reserved sqlite_ prefix instead without changing the streaming page census or its cancellation and snapshot boundaries. Add a file-backed regression that requires the omitted table's nonzero payload and subsequent growth to match an independent dbstat sum while still excluding sqlite_sequence. Preserve the concurrent CI repair.
|
Published a3654ac directly to this PR through GitHub ( Production bug fixed: the census excluded Regression added: the file-backed Rust test calls the production reader, requires a nonzero Executed SQL verification: using SQLite 3.46.1 on temporary files and SQL extracted from the candidate source, the old filter omitted 1,048,581 payload bytes; the new filter reports them and subsequently reports 2,097,162 bytes after another insert. The original fixture still has exact reference parity and 6,577 versus 19,043 VM steps. Interruption returns SQLITE_INTERRUPT, and retry returns the complete exact result. These are actual SQL checks, not a Rust/Bazel test pass. The uploaded source blob Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442890231 (queued at verification). Local Bazel and Rust toolchain are unavailable. Require the real current-head Bazel/Clippy/format results before merging. This is a bounded correctness fix, not evidence that the original 153 GB incident or #4099's 24-hour master acceptance is resolved. |
Refs #4099. Align the new user-table growth assertion with the exact format expected by the Bazel rustfmt gate; no behavioral changes.
|
Published The Repository gates log showed only a formatting difference in the new On The master baseline is unchanged at |
Refs #4099. The offline registry snapshot followed symlinked WAL/SHM members and could copy arbitrarily large foreign files into scratch. Refuse nonregular members before copying, and keep physical family byte counts from charging symlink targets outside the inspected profile. Add real offline-report and family-copy regressions for symlinked source members, plus partial-coverage and size assertions for user sessions. No operator data is copied or deleted.
|
Published 1e3d62f (Refs #4099), direct to this issue-specific PR. One-file update in New root-cause finding: the offline Fix: fail closed on every nonregular/linked snapshot member via Behavioral regressions: a real on-disk SQLite registry with a symlinked WAL and SHM, each targeting a sparse 512 MiB foreign file, must reject copying before a destination is created; the end-to-end offline report must return a typed Database failure for a linked registry WAL; and an offline report with a regular 4096-byte Independent checks: manually reproduced the original This is partial correctness only; #4099 remains open until 24-hour fresh-profile, multi-store bounds and daemon-owned reclamation are proved on |
|
Published The hosted Repository gates on Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42462849057 (queued at check). The previous run's Repository gates failed only on this formatting hunk; the Linux/Clippy jobs were still queued there at review time and are not evidence of a pass. There is no local Bazel executable or supplied checkout, so I do not claim a local Rust test pass. Require passing real nonvacuous The |
Refs #4099. The SQLite snapshot enumerated sidecars before capturing its validation generation. A writer creating a WAL in between could commit new project registrations that the report then misclassified as unregistered. Capture the generation first, then inspect only regular members and validate after copying. Add a deterministic on-disk WAL race regression: stale generations must fail closed, while a fresh generation preserves the committed project. Also fix the pinned Clippy collapsible-if failure in family sizing.
Refs #4099. The registry snapshot preflight rejected symlinks, but fs::copy reopened each checked path and could follow a link swapped in after inspection, copying foreign bytes into report scratch. Open through a held parent capability with no-follow leaf policy, then copy from that file handle. A real filesystem regression swaps a checked WAL member to a foreign link and proves the copy refuses without creating a scratch member. Preserve existing CI repairs.
|
Published New independently identified correctness/safety issue: The offline SQLite family copier checked each source member with The Linux filesystem TOCTOU mechanism was independently reproduced using a regular-file probe followed by symlink replacement: ordinary copy followed the foreign target while |
Refs #4099. A no-follow open still accepts a regular-file replacement between preflight and copy, potentially copying a huge foreign WAL before post-copy generation validation rejects it. Compare the opened descriptor against checked identity and length before creating scratch; bound the copy to the inspected length plus one. Add a filesystem regression for same-sized and oversized regular-file swaps without reading the foreign bytes.
|
Published Independent bug found: The preceding no-follow fix rejects symlink replacement but still opens an ordinary file swapped into the WAL/SHM pathname after the first metadata check. Post-copy Production repair: Preserve checked metadata for each SQLite main/WAL/SHM member; compare the opened, held file descriptor's identity and inspected length before creating the scratch member (Unix device/inode; non-Unix conservative size/time checks with existing generation validation). Cap each copy at the previously measured length plus one byte and reject a concurrent size change, rather than chasing an unbounded WAL tail. No dependencies, CI policy, thresholds, retention ownership, or production data were changed. Nonvacuous Unix regression: Replaces a previously checked, 4 KiB regular WAL with a different regular inode of equal size and again with a 512 MiB sparse file. It calls the production copy helper, requires a typed Independent filesystem reproduction: On Linux, |
Refs #4099. A rollback-mode writer may spill uncommitted registration pages into global.db while the undo journal remains. Copying only the main/WAL/SHM family would turn those pages into false registry truth. Fail closed on the rollback journal before writing scratch. Add direct and end-to-end real-SQLite regressions for refusal during an open transaction and correct registration after rollback. Preserve all prior snapshot generation and no-follow protections.
|
Published focused Root cause: the offline registry reporter copies only Published repair: refuse a present rollback journal before writing snapshot scratch and propagate the existing typed report failure instead of classifying stores using an unsafe registry. Two real-SQLite Rust behavioral regressions test the lower-level snapshot refusal (including no output) and the end-to-end The exact-head non-force fast-forward preserved |
Apply eight exact Rust 2024 rustfmt differences from hosted Bazel Repository gates. No behavior, schema, or test assertion changes.
Refs #4099 Charge nested WAL-looking host bundle and component filenames to their top-level directory bound, not an unrelated WAL key. Check exact physical bytes and named Doctor overages in a real filesystem regression, keeping genuine top-level WALs independent. Apply the pinned rustfmt corrections reported by exact-head CI.
Rename the predicate used by project kind accounting so it cannot shadow the SQLite path constructor used by offline registry snapshots. Restore Bazel Rust compilation without changing family classification, snapshot handling, or the existing filesystem-backed regressions. Refs #4099
The Doctor project overlay must only promote WALs rooted in the project store. Scratch files inside indexed artifacts, LCM payloads and response handles remain charged to the project, not unrelated WAL findings. Add a real profile filesystem census regression for exact bytes, project-kind attribution, and over-budget vs healthy Doctor evidence. Refs #4099
Only regular root-level WAL files and actual Grafeo WAL directories receive WAL bounds. Keep similarly named incident trees separately bounded and preserve genuine graph WAL pressure findings. Add real filesystem-backed Doctor and storage-kind regressions. Refs #4099
Refs #4099. A nested .wal payload is not a WAL family after root-only classification; test bounded evidence-key behavior directly.
Refs #4099. Preserve measured bytes but mark the missing store traversal unavailable and apply the unknown-file bound. Exercise normal and overbound replacements through the real footprint census.
Refs #4099. Test the degraded threshold using an explicit small limit while retaining the independent default-bound assertion.
Refs #4099. Preserve the concurrent freelist-aware retry fix and its regression. Add a real maintenance-registry journey proving unchanged work remains suppressed, logical deletion retries without compaction, old watermarks survive that retry, and subsequent growth is measured. Seed the existing deadline transition deterministically rather than relying on host speed. Keep production policy and measurement limits unchanged.
Review and published regression follow-up — Refs #4099Published 6179c8b on top of the concurrent production fix 95e7ad1, preserving that fix and its existing regression. The update used Root cause independently reproduced and production fix reviewed: timeout suppression compared allocated New behavioral coverage: Executed SQLite mechanism evidence (3.46.1, disposable filesystem only): before deletion: 2,054 allocated pages, 0 freelist pages, 2,054 dbstat rows, 8,388,736 payload bytes. After deletion: still 2,054 allocated pages and the same 8,413,184-byte file, but 1,988 freelist pages, 66 dbstat rows and 262,148 payload bytes. A subsequent insert produced 524,296 payload bytes without changing the physical file size. The old retry guard stayed suppressed; the corrected guard admitted the retry. This reproduction is not a repository Rust test pass. Validation: Linux fleet/Test Linux, Clippy and Dashboard passed on the preceding Keep #4099 open: fresh-profile 24-hour host acceptance and full master-proven reclamation are still outstanding. #4776 was verified open/draft and left untouched; no release, production, infrastructure, other branch or schedule changes. |
|
Validation update for published head 6179c8b: Repository gates in run #42723235152 has now failed, at The available log response is truncated before the failure diagnostics, and the connection does not expose a readable annotations/log-range action. I have not verified whether this is rustfmt or another policy failure and will not claim an exact formatting repair without those diagnostics. There is no local Bazel/Bazelisk or Rust toolchain in this execution environment. The new baseline-preservation/regrowth regression is published but not yet validated by repository Rust tests; this PR is not ready to merge on the evidence available here. The concurrent production fix |
Refs #4099. Match the three exact rustfmt hunks reported by Repository gates on the reclamation regression. No production behavior, test expectations, or CI policy changes.
|
Refs #4099: Published exact pinned-rustfmt repair |
|
Refs #4099 — exact-head Bazel qualification update (2026-11-10 03:30 UTC). New CI blocker: run #42727153286, Linux fleet failed 1 of 183 Bazel test targets. I downloaded the published Relevant #4099 Rust tests PASS on published head Sole Linux fleet failure: Hosted standalone Linux and Windows lanes are not complete at this checkpoint. No local Bazel/Bazelisk executable or checkout is present here; this is hosted Bazel Rust evidence, not a local run. #4794 remains open/unmerged and must not be qualified while required CI is red. #4099 remains open because daemon-owned reclamation and the specified 24-hour fresh-profile bound on |
Refs #4099. Directory enumeration follows symlinks when the walk root itself is linked, even though child entries use no-follow metadata. Check each pending directory before enumeration, keep its unreadable coverage explicit, and never attribute the linked target's bytes to the requested profile. Add a filesystem-backed regression: the linked profile reports a partial zero-byte floor without foreign evidence; the directly selected real root reports the exact 2048-byte overage. Preserve existing entry budgets and cancellation behavior.
|
Refs #4099 — published additional no-follow footprint fix Reproduced defect: Focused correction: require each pending directory to be a real directory via no-follow Nonvacuous regression: Publication reviewed against the GitHub commit diff: one descendant of Keep #4099 open and this PR unmerged: the required 24-hour fresh-profile host observation and complete daemon-owned reclamation on |
Refs #4099. The profile footprint census refuses a linked root, but the offline registry reader and per-project report still resolved child paths beneath that link and sampled foreign storage. Reject the linked profile authority before registry reads, paged reports or project sizing. Cover the actual offline and project report paths with a filesystem symlink and foreign payloads; explicitly selecting the real directory continues to report exact measured bytes.
Refs #4099. Seed a real foreign SQLite registry alongside the external project and sessions bytes in the linked-profile refusal journey. The explicit real-root control must report the registered project and exact payload bytes, proving the negative case is not vacuous.
|
Refs #4099 — published follow-up to the no-follow footprint fix on this existing issue-specific PR.
Independent local POSIX reproduction verified why a child-path Verified exact-old-head non-force fast-forwards Still partial: no master-proven daemon reclamation completeness or 24-hour fresh-profile bounded-growth acceptance; #4099 remains open. Root-link checks close stable linked roots and do not claim to solve adversarial rename/symlink races between metadata inspection and later path access. |
Scope
Refs #4099. This remains an issue-specific partial repair, not a claim that the deleted 153 GB profile has been reconstructed or that the 24-hour master acceptance has passed.
Measurement-path root cause and repair
The actual reader-worker
TableSizescommand joined onedbstatrow per database page tosqlite_schema, then usedGROUP BYto recover one total per table. That creates a temporary grouping sorter proportional to the number of pages being measured and consumes avoidable SQL VM work inside the existing measurement deadline. Large-store measurement was doing expensive page sorting in addition to the necessary page walk.The current reader initializes one total per user table from
sqlite_schema, then walksSELECT name, payload FROM dbstatonce on the same snapshot. It includes empty and rootless user tables, excludes index and internal pages from their totals, and returns an interruption or arithmetic error before publishing results. The earlier correlated per-tableSUMat8b403d1was an intermediate implementation.The existing registry fixes are preserved: the offline report distinguishes true
global.dbabsence from metadata/read errors and refuses registry symlinks rather than importing foreign registration identities into a profile whose census cannot account for those linked bytes.Behavioral regressions
Three new filesystem-backed tests call the production census implementation:
The earlier dangling/valid-external-registry behavioral tests remain in the PR.
Validation and review
9edee1472e796e6bde5bc1564b49a452c56e75d0before editing. Published blobs match the locally inspected source. Comparison against prior head43cef92shows only the production dispatch change and the new census/test module; the concurrent formatting fix is preserved.43cef92bc121f1bf9b451de1a3b92c8e40fd624f.8b403d1found only pinned-rustfmt differences in the new test module; the exact reported formatting corrections were published asd9559c8using a non-force fast-forward with expected old head8b403d1. The new-head CI run #42437543266 is queued; the old run's Rust jobs were cancelled by supersession, not passed. Required validation on the exact head:bazel test //crates/tracedecay-rusqlite-runtime:unit_test //crates/tracedecay-maintenance:unit_test --test_output=errors bazel build //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024 //crates/tracedecay-maintenance:rustfmt_check_2024Still required before closing #4099
This removes a demonstrated source of measurement work amplification; it does not make the physical page walk constant-time or prove that every large store fits its deadline. Per-family bounds, daemon-owned retention/reclamation and the fresh-profile 24-hour normal-use acceptance must still be demonstrated on
master. Keep #4099 open; keep Kev/decision-model work and mixed #4776 draft and excluded from merging.Current CI repair (679f636)
On the prior head
d9559c8, pinned Repository gates passed. Clippy failed onstorage_report.rs:2492withclippy::assert_is_empty(-D warnings): the added test usedassert!(absent.stores.is_empty()). Commit679f636replaces that with a typedassert_eq!(absent.stores, Vec::<StoreSizeReportEntry>::new()), retaining the failure check and adding value-level diagnostic output. Published via exact-headexpected_sha=d9559c8,force=false, with no concurrent commits overwritten.The earlier fleet Linux job failed with
UNAVAILABLE: io exceptionfrom the external Bazel remote executor (not a repository test assertion). The earlier Clippy job additionally observed that transient executor failure. Fresh hosted CI is pending on679f636. No Bazel executable or Rust toolchain is available locally, so the current Rust test and Clippy results remain unverified. Do not claim passing gates, merge, or #4099 closure until actual results and master acceptance exist.Partial WAL findings
The long-key WAL review and its duplicate are addressed in
55ef65ea44. When Doctor hashes a footprint reference, its coverage statement now retains the storage family and the unreadable-entry count. Both the measured-byte floor and the configured size bound remain visible.Native macOS Bazel passed all 10 footprint tests and the repository formatting check. The WAL regression walks a real regular segment and an unreadable directory entry, then verifies both a proven overage and a partial finding below the bound. Linux exercises a key over 1,024 bytes. The native macOS path uses a control character to trigger the same reference fallback within the filesystem path limit. Exact-head Linux remains pending.
Nested graph sidecar accounting
At
4b542edc22, the project Doctor bound and the physical storage report use the same classifier for the container file and its recognized sidecars. Both count a spill's nested files, including an orphaned spill directory, against its named container. Unrelated suffixes and unrelated directories with a.grafeoname stay outside the container bound. A dedicated WAL retains its separate bound.On the preceding head, replacing the flat orphaned spill fixture with a real
.grafeo.spill/scratch/inflightpath made the Doctor finding disappear. The adjusted case now reports the exact nested bytes and a degraded container bound. Both whole macOS Bazel unit suites passed on the repaired head, with 176 maintenance and 431 rusqlite runtime cases, plus repository rustfmt and strict Clippy on the maintenance test target. Current-head hosted CI, independent review and the separate 24-hour master acceptance for #4099 remain open.