Repository navigation
storage: the profile store grew to 153 GB with growth measurement unavailable #4099
Description
On 2026-10-16 ~/.tracedecay measured 153 GB before the owner had it deleted. Nothing reported the growth: tracedecay_runtime(doctor_report=true) listed storage.table_growth for both tracedecay.db and global.db as "table payload growth measurement is unavailable", so the doctor had no size finding at all.
What the folder held (listing taken before deletion; sizes per entry were not captured): projects/ with more than 30 proj_* stores, one per registered root, so each agent worktree under /fast/tmp that ran TraceDecay carried its own full code-graph index and sessions shard; lcm-payloads; user-sessions.db and user-sessions.grafeo with their WALs; global.db; maintenance, response-handles, hook_analytics.jsonl, host-bundle-stage, host-components. The live project_sessions shard for proj_a5b3d7e3ebe14ca7 had a 229 MB WAL whose checkpoint stayed pending through 245,264 retry wakes, and the host projection drain had been failing on a corrupt head since Oct 8 (separate lane). A past note records a staging container that grew to 8.6 GB and was loaded whole into heap.
Needed:
- Make size observable: fix the table-growth measurement, and make the doctor and
tracedecay storage reportshow per-store sizes (each project store, LCM payloads, Grafeo and WALs) with a finding when a store or the profile crosses a bound. - Retention and GC: project stores whose registered root no longer exists on disk (deleted worktrees); LCM payload retention; Grafeo WAL checkpointing; projection history; hook analytics rotation. Every GC goes through the daemon's own maintenance path, never a direct file delete.
- A regression check in the suites: a store that grows without bound under the ingest journey must trip the finding.
Acceptance: after a fresh tracedecay init, 24 hours of normal use on this host shows per-store sizes in the doctor, and no store grows past its bound without a named finding.
Activity
ScriptedAlchemy commentedon Oct 16, 2026
Status 2026-10-16. Landed or in the merge queue for this issue:
tracedecay storage reportshow it, Grafeo WAL segments are sized.hook_analytics.jsonlis truncated under its append lock once fully imported past 64 MiB.Remaining growth sources are assigned to the storage redesign cutover (
docs/plans/storage/README.md, #4118): retention for LCM payloads no summary covers and for unmounted session databases, observation projection history, the disposition ledger, generation staging release (step 9), the response-handle cap, hook events into a telemetry table (step 8), closing a lost store's lane instead of requiring a restart (step 7), and a 24-hour bounded-growth acceptance check on a fresh profile. This issue stays open until those land.Merge pull request #4125 from ScriptedAlchemy/fix/4099-grafeo-wal-che…
Merge pull request #4127 from ScriptedAlchemy/fix/4099-footprint-regr…
Merge pull request #4128 from ScriptedAlchemy/fix/4099-sidecar-retention
Merge pull request #4130 from ScriptedAlchemy/fix/4099-store-identity
fix(storage): avoid POSIX lock loss in daemon size reports
fix(storage): refuse symlinked footprint walk roots
ScriptedAlchemy commentedon Nov 7, 2026
Partial #4099 footprint-accounting safeguard staged on the held application PR #4776, not on master: 9d5693c (Refs #4099).
The read-only footprint walker rejected symlink children but followed a symlink at the supplied walk root, letting a foreign directory's files consume the scan budget and be counted against the requested store. The walker now requires each traversed directory to be a real directory; a directly symlinked root is reported unavailable, so Doctor coverage is partial rather than incorrectly healthy. A new Unix Rust filesystem regression covers two foreign files, no foreign byte charges, no consumed entry budget, and the partial-coverage result.
An independent Linux filesystem reproduction confirmed that the previous
read_dirfollowed the linked root and the no-follow guard refused it. Repository Bazel tests and native macOS coverage were not run locally; hosted CI for the new head was not yet available at verification. The guard does not claim to eliminate a concurrent path-replacement TOCTOU race, and explicitly configured symlinked profile roots require operator review of the partial finding.Keep #4099 open: the 24-hour bounded-growth/retention acceptance on a fresh host, especially large multi-store/profile measurements, is still unproven on
master. PR #4776 remains excluded from master merging because it contains Kev/decision-model work.fix(storage): reject unreadable profile registries
style(storage): format registry-report regression
fix(storage): refuse linked foreign registries in footprint reports
ScriptedAlchemy commentedon Nov 8, 2026
Published focused #4099 storage-report correctness repair: #4794 (current head
d2ea4f5,Refs #4099).The offline
storage reporttreatedglobal.dbas absent whenPath::exists()masked a dangling symlink or metadata error, so it could report project stores as unregistered and misdirect GC. It now distinguishes true absence from unreadability and refuses even a valid registry symlink: following that link would import foreigncode_projectsrows into the report while the footprint walk cannot attribute the linked file's bytes to this profile. Two Unix behavioral regressions cover the dangling link (and subsequent actual absence) plus a valid, externally linked SQLite registry. This is a one-file issue-specific PR, separate from held mixed #4776.Hosted Repository gates on the initial head found three rustfmt differences; the exact fixes were published in
6df1dce. The latest head has fresh CI queued (run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42433336288). Bazel/Rust tests were not run locally: no repository checkout or Bazel executable is available in this runner. No pass or merge is claimed.#4099 remains open. Published partial correctness does not establish the fresh-profile 24-hour, multi-store bounded-growth and reclamation acceptance on
master. No production, release, or other concurrent branch was changed.fix(storage): stream table growth census without page sorting
ScriptedAlchemy commentedon Nov 8, 2026
Published a substantive measurement-path correction in #4794, commit 8b403d1 (
Refs #4099). This goes beyond the earlier registry-path edge cases.Root cause addressed: the production reader's
TableSizesquery joined onedbstatrow per database page and then grouped those rows, creating a page-count-sized temporary sorter just to return one total per table. The replacement streams a correlated payload sum per table and sorts only completed table totals. It keeps page-modedbstatso interruption is observable between pages; admission, snapshot ownership, deadlines and typed failures are unchanged. It does not replace physical footprint accounting with payload bytes or increase a timeout.Published regressions: real filesystem-backed SQLite overflow payloads, WITHOUT ROWID and empty/rootless tables; exact equality to an independent page-level reference; insert growth; logical deletion versus physical compaction; actual VM-work reduction under the same fixture; and interruption followed by a successful exact retry. Restoring the old grouped query makes the work-regression assertion fail. No test relies on source text or a wall-clock speed threshold.
Executed mechanism evidence, not a Rust-test claim: SQLite 3.46.1, 1,040-page fixture, identical initial payload totals with 19,043 VM steps before / 6,577 after. The events payload measured 4,194,384 bytes, increased to 4,456,533 after insertion, and fell to 262,149 after deletion. Physical pages stayed at 1,105 until vacuum of the disposable fixture reduced them to 78 without changing the remaining payload measurement. Interruption/retry checks passed. Published source blobs were matched against the locally inspected files and the worker diff independently reread.
The concurrent formatting commit
43cef92was preserved, and publication used a non-force fast-forward with that exact expected old head. Current master remains91c72ab; #4776 is still draft and was not modified or transplanted. Other owner schedules and release #4763 are unchanged.Validation outstanding: this runner has no Bazel executable/toolchain. Hosted CI https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435823396 is running/queued on
8b403d1; no repository Rust compilation/test/format pass is claimed. Validate//crates/tracedecay-rusqlite-runtime:unit_test,//crates/tracedecay-maintenance:unit_test, bothrustfmt_check_2024targets, and the existing CI gates before merging.Keep #4099 open. This removes a demonstrated measurement-work amplification, not the necessary O(pages) scan. It does not yet prove all large-store measurements succeed, the full per-family retention/reclamation bounds, or the fresh-profile 24-hour normal-use acceptance on master.
style(storage): align census regressions with pinned rustfmt
75 remaining items
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4099 — published named project-store partial-coverage repair on dedicated PR #4794: source and regression
a1513efplus format follow-up22bfc69.Root cause: The physical footprint walker correctly refused a symlinked project-store directory and reported it unavailable, but the classifier required at least three path components to recognize a project store. The unavailable root itself has only two (
projects/proj_blocked), so Doctor assigned its partial coverage to an unrelated genericprojectsentry and omitted the affected project's named finding entirely. The same misattribution applies to unreadable or invalid project-root entries.Repair: Classify the project-store root itself as belonging to its project, even when no descendant can be walked. This does not follow the symlink, count foreign data, or mistake unknown coverage for zero-byte healthy storage. Existing per-project and profile bounds remain unchanged.
Behavioral Rust regression: Real filesystem fixture with an actual 2 GiB sparse foreign directory, a symlink replacing one registered project's store, and an unrelated healthy 11-byte project. Requires exact 11-byte owned profile total; 1 unavailable entry; two named project entries, not
projects; registered-root status preserved; the blocked project Partial, unaffected project HealthyCompleteCoverage, and profile Partial. The old classifier fails becauseprojects/proj_blockedis absent. An independent disposable-filesystem reproduction confirmed old vs corrected key selection; not a Rust/Bazel test pass.GitHub compare confirms two descendant commits, one changed file (+64/-1), exact-old-head non-force fast-forwards. PR #4799, held mixed draft #4776, Kev/decision-model branches, release #4763 and master were not changed. New-head hosted CI is queued; Bazel/Rust/Clippy and pinned formatting are not yet confirmed passing for this head. No local Bazel or checkout is available.
Keep #4099 open: full daemon-owned reclamation and the 24-hour fresh-profile bound/retention acceptance are still unproven on
master.style(storage): satisfy pinned footprint rustfmt (Refs #4099)
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4099 — published current-head pinned-rustfmt repair to the existing issue-specific PR #4794: commit
fb4b65f.The preceding CI run #42687881011 failed Repository gates at
//crates/tracedecay-maintenance:rustfmt_check_2024on exactly one assertion in the new symlinked-project-store regression (retention/footprint.rs:1173). Applied the exact formatter-requested multiline form without changing its semantics. Eight unrelated repository test targets passed before that formatter failure; Linux fleet and Clippy remained in progress when inspected. The dependent Feature gates failure follows the Repository gates failure.Verified
22bfc69->fb4b65fis exactly one descendant commit changing onlycrates/tracedecay-maintenance/src/retention/footprint.rs(+4/-1); published usingforce=false,expected_sha=22bfc69...and no peer branch updates. Master and release #4763 were not changed; mixed #4776 remains draft and held. The runner has no Bazel/Bazelisk or project checkout, so repository Rust tests/Clippy/new-head formatting remain unverified until hosted CI completes.Keep #4099 open: full daemon-owned reclamation and the mandatory 24-hour fresh-profile bounded-growth acceptance on master remain outstanding.
test(storage): keep hashed WAL diagnostic test valid
fix(storage): expose replaced project-root files in doctor
test(storage): exercise replacement bound without large disk allocation
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4099 — published CI regression repair plus reviewed project-root observability fix on issue-specific PR #4794 (exact-head, non-force fast-forwards, no other branches changed).
1825f4arepairs the real hosted Linux failure inretention::footprint::tests::hashed_wal_reference_keeps_overages_and_partial_coverage(CI #42696895313: 184/185 maintenance tests passed). The old fixture expected a nestedsessions.walpayload to be a WAL family, which contradicts the already-correct root-only WAL classification. The replacement directly exercises the production Doctor finding/hash/partial/overage path for an oversized identifier, while existing real-filesystem tests verify nested WAL-looking payload ownership. This is a diagnostic-level unit regression, not a claim that an impossible >1024-byte WAL family path can be created on disk.7e02189fixes the new review finding: when a regular file replacesprojects/proj_a/, its measured bytes stay in the profile total but the missing directory traversal is now unavailable for that named project instead of falsely healthy. Such a file uses the normal unknown-entry default bound (1 GiB), not the 10 GiB valid-project allowance; explicit owner overrides remain authoritative. A real TempDir regression checks exact 18-byte physical totals, a partial replaced-project finding, a healthy unaffected project, and degraded state under a deliberately small overage bound.f0d9914keeps that overage fixture cross-platform without creating a 1 GiB sparse file on Windows.Independent GitHub comparison verifies three descendants of
fb4b65f, a single changed file (retention/footprint.rs), and no overwritten peer head. New-head CI #42710315413 is queued; no Bazel/Rust pass on the new tip is claimed. The prior head also had independent Linux runtime/LSP timeout failures; they are not silently assigned to the storage fix.Keep #4099 open: master does not yet prove daemon-owned reclamation of every growth family or the 24-hour fresh-profile normal-use acceptance. Held #4776 and Kev/decision-model lanes remain draft and unmerged; release #4763, production/infra and other schedules untouched.
fix(storage): retry table census after freelist reclamation
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4099 — published freelist-reclamation growth-measurement repair on the existing dedicated PR #4794: commit
95e7ad1.Root cause: After a prior per-table
dbstatcensus exceeds its deadline, the daemon suppresses retries for 24 hours while SQLite's allocatedpage_countstays at least as large as its failure mark. But normal retentionDELETEreclaims overflow pages into SQLite's freelist without shrinkingpage_count, so the next read may be far cheaper and still gets incorrectly skipped. This leaves Doctor table-growth measurements unavailable long after successful reclamation.Fix: Compare the number of live/non-freelist pages (
page_count - freelist_pages) against the most recent failed walk's live-page count. Preserve the original 24-hour age-out, smaller-work retry behavior, explicit overflow failure, cancellation, error propagation, and restart re-probe; do not change GC or budgets.Nonvacuous filesystem-backed Rust regression: The production guarded telemetry port and real database runtime insert eight 1 MiB blobs, force a fresh timed-out-walk marker, require initial
ExceedsMeasurementBudget, delete the blobs only in a disposable test database, require unchanged allocated page count with increased freelist pages, then require the same port to retry and establish the table-growth baseline. The old allocated-page check fails that last assertion. An independent SQLite 3.x reproduction showedpage_count2053 -> 2053,freelist_count0 -> 2051, anddbstatpages 2 after deletion. That confirms the mechanism, not a repository Rust/Bazel pass.Publication: Exact-old-head non-force fast-forward
7de8bcd->95e7ad1, one descendant commit, onlycrates/tracedecay-maintenance/src/telemetry.rschanged. The new-head CI #42722485068 is queued; Bazel Rust tests, pinned formatting and Clippy remain unverified on this head (no local Bazel executable/checkout). Held #4776 remains draft and unmerged, no peer-owner branches or release #4763 touched.Keep #4099 open. Complete daemon-owned reclamation and the required 24-hour fresh-profile bounded-growth normal-use acceptance have not been proven on
master.test(storage): preserve growth baselines through reclamation
style(storage): apply Bazel-pinned Rust formatting
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4099 — published exact Bazel-pinned Rust formatting repair to the existing dedicated PR #4794: commit
aaf3e6f.Read the complete log for Repository gates #42723235152 on
6179c8b. Contrary to the earlier partial log response, the precise failure is now confirmed: Bazel//crates/tracedecay-maintenance:rustfmt_check_2024rejected exactly three formatting hunks: the multiline live-page assertion andmatches!intelemetry.rs, and thelet ... else.unwrap() else {layout in newtelemetry/reclaim_tests.rs. All eight other repository-policy tests passed in that job.Applied exactly those formatter-requested changes, with no production semantics, test assertions, bounds, build configuration or CI-policy changes. Independently compared
6179c8b->aaf3e6f: exactly one descendant commit changing these two Rust files (+9/-4), and published viaforce=falsewithexpected_sha=6179c8bb711c2e3e4f8941fcb4398a7831e2244b. The concurrent production fix and regression at95e7ad1are retained; no peer branches, held drafts, release, operator data, or schedules touched.Exact-head CI #42727153286 was queued at verification. Bazel is absent locally, so the new Rust regression and Clippy are not yet proven passing on this head. The 24-hour fresh-profile host acceptance and all required master-proven reclamation remain outstanding; keep #4099 open. #4776 was verified open and draft and left untouched.
fix(storage): refuse linked profile roots in footprint census
fix(storage): refuse linked roots before report sampling
test(storage): prove linked roots refuse real registry sampling