Sealed code-graph generations (tracedecay.sealed/) are never collected; busy project store grew to 56 GB over a 10 GB bound

tracedecay version

1.0.0-beta.92+c443aaa71d48457d07faa2737fac42164255371a (upgraded from 1.0.0-beta.87 during the observation window; the growth was seen on both)

Operating system

macOS 26 (Darwin 25.6.0), Apple Silicon (aarch64)

Installation method

Release binary on the beta channel, updated with tracedecay upgrade. The daemon runs as the managed launchd service.

What happened?

<store>/tracedecay.sealed/ grows without bound on a busy project. Old sealed code-graph generations are never removed, and nothing reclaims them short of tracedecay projects forget + tracedecay init.

One project store reached 56 GB against its 10 GB bound (doctor reported 76.6 GB). The profile total was 114 GB against its 64 GB bound.

Breakdown of that one project store before the reset:

Path	Size	Notes
`tracedecay.sealed/`	24 GB	73 generation directories, ~670–815 MB each, written between Oct 12 and Oct 29. None was ever removed.
`code-index-v1/`	15 GB	11 GB `code-text-artifacts-v1`, 2.4 GB `code-generation-segments-v1`
`sessions.db`	16 GB	transcripts (expected, out of scope here)
`tracedecay.graph-replay/`	749 MB

New generations were written at 2–15 per day:

Oct 12: 2   Oct 13: 10   Oct 14: 9   Oct 15: 2   Oct 16: 7   Oct 17: 2
Oct 19: 15  Oct 20: 13   Oct 21: 4   Oct 22: 7   Oct 29: 2

After projects forget and a fresh init on beta.92, the new store wrote 2 sealed generations (820 MB + 721 MB) within ~20 minutes. Whether those get collected is not yet known; see "Additional context".

The project is a large monorepo: about 19k files and ~110 MB of indexed source (Python, Rust, TypeScript, Go). It has ~170 linked git worktrees, and before cleanup it had 56 tracked branches. Many agents commit to it throughout the day, so rebuilds are frequent. That explains the rate. It doesn't explain why nothing is ever collected.

Expected behaviour

Superseded sealed generations are collected once no reader holds them, so tracedecay.sealed/ stays near one or two generations per live scope. A store should stay within its configured bound, or doctor should name a supported command that reclaims the space.

Steps to reproduce

Enroll a large, frequently-committed repository (thousands of files, many commits per day, several tracked branches or linked worktrees).
Let the daemon run for several days.
Watch ls <profile>/projects/<id>/tracedecay.sealed | wc -l and du -sh on that directory. The count only goes up.
tracedecay doctor reports the store over its bound, but branch gc, branch remove, and the daemon's automatic retention ticks don't shrink tracedecay.sealed/.

Error output

The daemon log covers only the last ~11 hours because it rotates at 32 MB. In that window, every retention event concerns code-index-v1; no retention event mentions tracedecay.sealed:

event=retention_degraded pass=code_generations failure=generation_store_busy                       (x4)
event=retention_degraded pass=code_index_scopes failure=scope_recovery_failed error="code-generation retention deferred: generation store is busy"   (x3)
event=retention_degraded pass=code_generations failure=retention_plan_failed error="code-generation retention refused unsafe state: text-artifact retention recovery would remove a durable descriptor target"   (x2)
event=retention_maintenance_tick succeeded=false outcome=retry ... reclaimed_bytes=0
event=retention_code_generations store=code-index-v1 bytes_reclaimed=308383744 generations_collected=0 text_artifacts_collected=1
event=retention_code_index_scopes store=code-index-v1 live_scopes=47 collected_scopes=1 ...
event=daemon_shutdown outcome=maintenance_task_aborted owner=retention_tick reason=cooperative_join_exceeded_abort_deadline

doctor (beta.92) before the reset:

✘ storage: project store projects/<id> (root present); graph container 22.4 MB, replay pool 748.6 MB: 76.6 GB of 10.0 GB bound
✘ storage: WAL .../code-text-artifact-staging-v1/.text-artifact-<hash>.staging-wal: 565.6 MB of 128.0 MB bound
✘ storage: profile total: 114.0 GB of 64.0 GB bound
! storage: code-index scope roots whose project root no longer exists hold bytes no scope-local retention pass can reach (... stranded-scopes-16 ...)

tracedecay storage report was killed (exit 137, likely OOM) on this store, so no dry-run plan is available.

Component

Daemon / storage retention (tracedecay-maintenance)

Additional context

Notes for the agent picking this up

Goal: sealed code-graph generations under tracedecay.sealed/ are collected like other superseded generations, and a busy store stays within its bound without manual projects forget.

Leads (from code search on the default branch, not verified against the running build):

crates/tracedecay-maintenance/src/retention/storage_report.rs defines SEALED_GRAPH_DIRECTORY = "tracedecay.sealed" and counts it as StoreKindBytesV1::sealed_graph. That was the only place code search found the constant. It is measured, but no retention pass that collects it turned up. First question: does any pass delete sealed generations, and is it scheduled?
crates/tracedecay-maintenance/src/store_maintenance/mod.rs emits retention_code_generations. Check whether the code_generations pass is meant to cover tracedecay.sealed/, or only code-index-v1/.
The retention_plan_failed ... would remove a durable descriptor target refusal aborts the whole code_generations plan. If one unsafe text artifact blocks every collection for the store, that alone can stall reclamation. It may be worth making the refusal skip that artifact rather than the entire plan.
generation_store_busy deferrals and cooperative_join_exceeded_abort_deadline on shutdown suggest that on a busy store the pass rarely gets the lease. A pass that can never acquire the lease under steady write load is starvation, not safety.

Acceptance criteria:

A regression test builds N > 3 sealed generations for one scope, publishes the latest, releases readers, runs retention, and asserts that only the live (and at most one previous) generation remains under tracedecay.sealed/.
A regression test shows that one unsafe or unresolvable text artifact doesn't stop collection of the other superseded generations in the same store; the unsafe one is reported and skipped.
Under sustained write load, retention makes forward progress: bounded deferrals, then it runs.
doctor's over-bound finding names a supported reclaim command when automatic retention can't fix it.
tracedecay storage report completes on a store with ~70 sealed generations and ~15 GB of code index without being OOM-killed, or bounds its memory and reports partial coverage.

Out of scope: session transcript growth (sessions.db, user-sessions.db) is a separate retention question.

fix(retention): retry failed graph staging releases

fix(storage): bound sealed census to its own directory

fix(graph): attribute shared sealed bytes to live heads

Partial progress on sealed-generation footprint accounting (not a collection fix): a121f854, staged on the sole application integration PR #4776, not on master. Layered generations hard-link base files, while the census previously visited directories in arbitrary read_dir order and charged each inode to the first directory, even unrecognized crash debris; that could wrongly mark a live head's physical bytes as retired or omit them entirely. The fix classifies directory receipts first and charges live heads before superseded, staging and unknown aliases. A deterministic Unix Rust regression explicitly visits unknown/staging/retired paths before the live head and tests the no-head case.

Validation: An independent Linux filesystem analog exercised all 24 traversal permutations (old counting misattributed the head in 16; head-first in 0). Repository Bazel tests and native macOS acceptance have not run locally; exact-head hosted CI is pending at run 42233694254. The PR is held from master merging because it includes excluded Kev/decision-model work.

Keep #4616 open: this corrects census attribution only. The N>3 retirement journey, unsafe-artifact isolation, progress under sustained writes, supported doctor reclaim guidance and large-store OOM-bounded reporting remain unproven.

fix(storage): preserve newer and unknown sealed artifacts

fix(graph): refuse symlinked sealed roots during cleanup

docs(graph): keep census contract on its entry point

fix(graph-db): require bound sealed census receipts

fix(storage): preserve unknown sealed forms on format cleanup

fix(graph): refuse unknown sealed forms in live census

Partial sealed-census correctness repair staged on #4776, not on master: 968c505 (Refs #4616).

The census previously treated a parseable, correctly named receipt as a live head even if its sealed form was unknown or said layered without a base. Those artifacts could inflate live counts and claim shared hard-linked bytes. The change reuses one recognized-form/base policy in census and obsolete-format cleanup; a deterministic Unix Rust filesystem regression adds both bad receipts, a foreign payload, and hard-link aliases and asserts the live head keeps its bytes.

Validation status: an independent four-case Python receipt-classification analog passed, not a repository Rust/Bazel test. Bazel and native macOS execution were unavailable locally. Exact-head hosted CI is pending; no green claim. The issue remains open: generation count N>3 actual reclamation, unsafe text-artifact isolation, bounded progress under sustained writes, supported doctor reclaim guidance, and OOM-safe large-report acceptance are not proven on master. #4776 remains on an explicit master-merge hold because of excluded decision-model changes.

fix(graph): exclude unsupported graph formats from sealed census

fix(retention): isolate unsafe unreferenced artifact entries

Partial retention-starvation repair staged on integration PR #4776, not on master: 6cd9a91 (Refs #4616).

In plan_collectable_text_artifacts_budgeted_page_cancellable, a symlink or directory in the unreferenced shared/staging artifact inventory previously returned UnsafeState and vetoed all otherwise verified generation/text-artifact retirement. The patch reports and preserves the unreferenced nonregular entry, then continues the bounded scan. Durably referenced descriptors and active staging remain fail-closed. New Unix Rust filesystem regressions require reclamation of four superseded generations plus a verified orphan in the presence of an unrelated symlink/directory, preservation of foreign bytes and the live descriptor, and continued refusal when a referenced descriptor becomes a symlink.

Validation: reviewed two-file exact commit; expected-old-head nonforce fast-forward independently verified. Independent Linux filesystem/policy probe only; repository Bazel and native macOS tests not executed locally. Exact-head hosted CI is queued, not passed.

Keep issue open. This addresses one unsafe-artifact isolation path, not the complete N>3 sealed retirement journey, bounded progress under sustained writes, supported doctor reclaim guidance, or ~70-generation large-store OOM/partial-coverage acceptance. It is on a PR held from master merge because #4776 also contains user-excluded Kev/decision-model changes; staging does not prove default-branch resolution.

fix(graph): bound sealed receipt reads before cleanup

New #4616 master-acceptance blocker confirmed during this review: Doctor reclaim PR #4799 is marked merged, but its base was codex/cold-git-sync-diagnostics, not master. The resulting merge commit c1d1c901 is divergent from current master aa03f27e, and the actual current master crates/tracedecay-maintenance/src/retention/footprint.rs does not contain registered_project_reclaim_hint or its supported tracedecay --dry-run projects forget / --yes commands. Thus #4799's merged badge is not evidence for #4616 acceptance criterion 4 on master.

The active storage PR #4794 also owns retention/footprint.rs, so this lane deliberately did not paste #4799 onto or overwrite that concurrent file. Coordinate the focused Doctor cutover with that owner and prove the real over-bound finding and tests on master before closing #4616. Protected draft #4776 and release #4763 remain untouched.

fix(graph): refuse missing registered sealed retirement

style(graph): align retirement error mapping with rustfmt

test(graph): exercise relocated sealed paths on Unix

Refs #4616 — published three nonforce, exact-expected-old-head fast-forward commits to the existing issue-specific PR #4819. Verified final remote head b9798d71, three commits ahead/zero behind, only one graph-db Rust source/test file changed (+95/-3).

6027cbac fixes a physical-retirement false success: when a registered sealed reader's root or generation directory disappeared before the held-capability cleanup opened it, Ok(false) was incorrectly mapped to successful retirement. If moved aside, its bytes were still durable, but the reader was uninstalled and the parent could mark it collected. For a registered expected digest, absence now refuses with a typed error and retains retry ownership; disk-only duplicate retirement remains idempotent.
9c3dfc48 normalizes mapping layout for rustfmt; b9798d71 restricts the physical directory relocation regression to Unix.
The new real native behavioral regression builds a verified sealed store and registers it in a persistent parent. It independently moves the entire sealed root and the generation child, requires each retirement to refuse while retaining the exact reader and receipt bytes, restores each directory, then proves successful physical retirement and idempotent disk-only retry. The previous code would falsely return success on the first attempt.
Independent source review published. Exact-head hosted CI #42752083614 is queued, not a passing Bazel/native result; this environment has no checkout or Bazel. No Cargo was used.

#4616 remains OPEN. Complete automatic N>3 retention with sustained writers, durable ancestor identity, Doctor reclaim integration and incident-scale report behavior are not jointly proven on master. An additional current review flags the unmeasured 65,536-inode census threshold, which must be justified or redesigned before final acceptance. Doctor #4799 and writer-admission #4813 merged into prerequisite branches rather than master; active storage owner #4794 overlaps Doctor and was not touched. Protected mixed draft #4776, Kev/model PRs, release #4763, other owner branches and task schedules were left unchanged.

New #4616 blocker confirmed by source-path audit on master d260a49f and issue PR #4819 b9798d71 (not a published fix or a native test pass):

GraphDbRegistry::retire_superseded_projection_replays_inner returns SupersededReplayRetirement::default() as soon as verified_head(projection) returns None, before it enumerates retired_cleanup_page. The only production finalization path for a durable retired replay tombstone is finish_retired_replay later in that same routine. Consequently, if the scope loses its verified head after a relational replay has been retired but before physical sealed directory removal / tombstone finalization succeeds, later sweeps of that headless projection skip the existing tombstone forever, leaving eligible derived graph bytes stranded even when no reader remains. Current master early return.

Focused regression needed: publish multiple real sealed generations, induce interruption after the first durable retire_replay tombstone but before native cleanup, remove the projection's verified head, and run the real registry retirement again. Require completion of only the preexisting tombstone, physical reclamation of its verified sealed artifact, and preservation of non-tombstoned generations and foreign bytes. No-head state must not authorize newly retiring arbitrary replay records.

The active #4508 owner PR #4800 already changes crates/tracedecay-graph-db/src/registry/publication.rs; this #4616 lane deliberately did not edit or overwrite that concurrently owned file. Review/reconcile this separate retirement branch after the owner releases it. Existing #4819 changes and user-protected draft #4776, model work, release #4763, other branches, CI and schedules remain untouched. No Cargo or Bazel run was performed (this session has no repository checkout or Bazel). Keep #4616 OPEN: this does not establish end-to-end retention/fairness/Doctor/incident-scale acceptance on master.

fix(graph): import sealed retirement verifier in native test

Refs #4616 — published commit 1f3e3364 on the existing issue-specific PR #4819. GitHub verified an exact-old-head b9798d71, force=false fast-forward, one commit ahead / zero behind, with only the nested graph-db test import reformatted (+3/−3). No other branch or protected PR changed.

Concrete exact-head CI blocker corrected: Linux Bazel / Clippy run #42752083614 failed both //crates/tracedecay-graph-db:unit_test__binary and strict Clippy with Rust E0425 (sealed_store.rs:4201). The latest registered-reader relocation regression called retire_sealed_directory_checked without importing it from its parent module. The commit imports the existing production helper; no test assertions, deletion behavior, or CI gates were weakened. The regression still requires refusal with relocated durable bytes, then successful removal after restoration and disk-only idempotence.

Independent exact-head review published. New CI #42764317492 pending, not a pass. No local checkout/Bazel was available, no Cargo executed.

#4616 remains open: scheduled N>3 retirement during sustained writers, stable ancestor/staging ownership, integrated Doctor reclaim (PR #4799 merged to a prerequisite branch, not master), incident-scale memory-safe reporting, and all acceptance proven on master remain unresolved. The reviewer-reported unmeasured 65,536-shared-inode cap also remains under review. Draft #4776, Kev/decision-model work, release #4763, concurrent owner tasks and existing schedules untouched.

test(graph): align graph-db unit tests with sealed retirement contracts

fix(graph): release fully observed hard-link census identities

Refs #4616 — published 6939336a to the existing focused PR #4819, using expected_sha=736779337edbefd5a640c37c1dff432e5079c2ec and force=false. GitHub confirms one commit ahead, zero behind, and only crates/tracedecay-private-fs/src/capability_dir.rs changed (+59/-6). Other branches, protected drafts, release, CI configuration, and schedules were untouched.

New fix: the bounded sealed-store census was retaining hard-link inode identities even after visiting every alias. Many independent small hard-link groups could therefore trigger a false OutOfMemory / partial-coverage refusal despite a constant number of unresolved groups. The walker now decrements the initial filesystem link count, evicts fully observed groups, and preserves the existing live-head priority transfer and fail-closed bound for external/unresolved aliases. A native Unix regression constructs four independent two-link groups in separate directories, verifies that an actual census with a one-identity budget succeeds with 21 uniquely charged file bytes, and checks both links' data remain intact; the preceding implementation fails at group two. The previous external-alias refusal regression remains.

Independent exact-head review published. Exact-head CI #42767614226 is queued, not passed. No local Bazel executable/checkout exists in this session, no native test pass is claimed, and no Cargo was invoked.

#4616 remains OPEN: the 65,536-inode production cap still lacks measured justification; automatic N>3 physical retirement under sustained writers, owner-bound store ancestors, Doctor reclaim on master, and incident-scale complete-or-partial OOM-safe reporting have not all been proven on master. Do not merge/close on this partial fix; mixed draft #4776 and Kev/decision-model PRs stay draft and unmerged.

test(graph): keep census scheduling private and preserve layered proof

test(fs): import Unix inode metadata in census test module

Refs #4616 — published three exact-old-head, force=false fast-forward commits to existing focused PR #4819: cc772b30, 80977b3e and 6ac18619. GitHub verified the final head 6ac18619. Only the two issue-owned Rust source/test files changed.

Review blockers addressed: tracedecay-private-fs had exposed a feature-gated test_support::census_in_entry_order API solely for graph testing, against root AGENTS.md's rule banning test-only production ports. The port is deleted. A private native filesystem unit regression instead drives the actual census core with real hard links and deterministic late publication, covering both initially single-linked and fully observed two-link groups, exact physical byte deduplication, live-head priority and foreign-byte preservation. Real graph-db layered-publication regressions remain and invoke the production public census; Unix metadata traits now use test-module-scope imports. The additional third commit restores the integration tests' required head_count=1, superseded_count=1+initial_layers, and unrecognized_count=0 assertions after reviewer feedback.

Exact-head independent source review and first two-commit review published. Exact-head CI #42777493212 is queued, not passing evidence. No local Bazel checkout/native runner exists; no Cargo was run.

Keep #4616 open. The 65,536-inode limit lacks measured justification; scheduled N>3 retirement under sustained writers, owner-bound store ancestors, Doctor reclaim guidance on master, large-store complete/partial reporting and combined master acceptance remain outstanding. Draft #4776, Kev/model PRs, release #4763 and other owners' branches untouched.

test(graph): assert layered census generation counts

Navigation Menu

Repository navigation

Sealed code-graph generations (tracedecay.sealed/) are never collected; busy project store grew to 56 GB over a 10 GB bound #4616

tracedecay version

Operating system

Installation method

What happened?

Expected behaviour

Steps to reproduce

Error output

Component

Additional context

Notes for the agent picking this up

157 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Sealed code-graph generations (tracedecay.sealed/) are never collected; busy project store grew to 56 GB over a 10 GB bound #4616

Description

tracedecay version

Operating system

Installation method

What happened?

Expected behaviour

Steps to reproduce

Error output

Component

Additional context

Notes for the agent picking this up

Activity

ScriptedAlchemy commented on Nov 6, 2026

ScriptedAlchemy commented on Nov 6, 2026

ScriptedAlchemy commented on Nov 6, 2026

157 remaining items

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions