[ScriptedAlchemy]

Summary

TD could store the same tool result in both replay metadata and its compressed canonical body, retain unreferenced canonical bodies, and move compactable history into uncompressed files. Metadata-only updates also rewrote the search index, while bounded incremental vacuum reclaimed only one page regardless of its requested budget.

This change removes those sources of storage and write amplification through the existing ingestion, registered-store retention, and authorized writer paths.

Changes

Keep one recoverable copy of tool evidence

• New ingestion omits replay content only when it is a plain string exactly equal to the canonical raw body. The existing verified reader reconstructs it for replay. Structured content, media and external-payload placeholders retain their current behavior.
• A resumable historical pass applies the same rule to existing metadata: at most 48 rows and 4 MiB of metadata/stored/hydrated input per step. It verifies the canonical body and sanitizer receipt, resanitizes the remaining metadata, and uses an exact row-state compare-and-swap before commit.
• Canonical-body collection tracks both raw-message and observation owners. Transactional writer triggers are installed before bounded bootstrap; deletion starts only after all captured owner/body frontiers complete. Ownership, candidate queues and bootstrap progress are durable, and each step validates the installed schema before deleting anything.

Compress retained history without creating new payload files

• Small historical retention_offload files return to ordinary inline storage. Larger qualifying files use the existing compressed canonical-body representation when that representation is smaller.
• Future retention compaction creates no new uncompressed external payload files. Non-beneficial direct-inline bodies remain intact.
• The savings decision includes the marginal compressed BLOB, exact UTF-8 snippet bytes and a conservative 1 KiB record/reference allowance. Reusing an existing body is evaluated and verified inside the writer transaction.
• Each lossless storage pass is bounded to 500 candidates and 8 MiB of body/envelope input, with a 4 MiB per-body/envelope limit. A new restoration cursor revisits files the earlier small-file pass skipped.
• Source ownership, hashes, receipts, metadata, timestamps and row state are checked before commit. Existing external files remain until the existing owner-aware payload GC and grace period permit deletion.
• Verified representation changes preserve summary convergence state. Only the exact validated invalidation trigger is suspended inside the authorized transaction and restored before commit; other privacy, ownership, payload and FTS triggers remain active.
• Raw readers fetch canonical BLOB columns only for actual canonical-backed rows, avoiding unrelated BLOB hydration.

Reduce background work and make reclamation accurate

• The FTS update trigger reindexes only when an indexed value or row ID changes. Admission refreshes this derived trigger transactionally, and the admitted-schema fingerprint invalidates cached older definitions.
• Incremental vacuum drains SQLite's result rows, allowing the configured page budget to take effect. Writer authority and rollback behavior remain intact.
• Observation retention measures UTF-8 bytes and accounts for both replacement provenance markers. Logical content reduction, deleted compressed BLOB bytes and physical disk reclamation remain separate metrics.
• Bounded storage work resumes on the existing short maintenance cadence, including across store round-robin windows. Continuation honors retention settings and disables the age-based raw-drop policy; normal retention windows, payload-file GC grace and physical-compaction scheduling remain unchanged.

Dependency integration

Includes #4721 through its commit ancestry. Grafeo's allocation APIs are maintained directly in the fork's main branch, pinned by the lockfile to 5116be3c; the local Bazel patch is removed. There is no transitive dependency churn.

The replay-token-accounting changes in #4712 and Hermes integration changes in #4711 remain independent.

Regression coverage

Added or extended regressions cover exact replay recovery, Unicode and representation-cost boundaries, canonical sharing and corruption, owner-aware GC bootstrap and races, cursor/budget continuation, stale candidates, receipt/authority failures and transaction rollback. Additional cases cover unchanged summary convergence during verified restoration, ordinary subsequent invalidation, FTS content/NULL/row-ID updates, populated incremental vacuum and rollback, and continuation without raw-drop work.

Validation

Passed:

• CARGO_BUILD_JOBS=1 cargo check --profile bounded-test -p tracedecay-lcm --lib --tests --locked.
• CARGO_BUILD_JOBS=1 cargo check --profile bounded-test -p tracedecay-global-db --lib --tests --locked.
• CARGO_BUILD_JOBS=1 cargo check --profile bounded-test -p tracedecay-maintenance --lib --tests --locked.
• CARGO_BUILD_JOBS=1 cargo check --profile bounded-test -p tracedecay-rusqlite-runtime --lib --tests --locked.
• Rust formatting and git diff --check on the storage changes.
• Independent review of ownership, schema admission, CAS/receipts, rollback, representation costs, summary convergence, GC and maintenance continuation, including the composed publication tree. The published tree was checked against the frozen working blobs.
• Supplementary SQLite 3.53.1 probes using the production FTS SQL: metadata/no-op updates make no FTS shadow writes; indexed content/NULL/row-ID changes and trigger replacement rollback remain correct. Incremental vacuum requested for eight pages reclaims eight after draining, versus one with the old single-step behavior; rollback reclaims none.
• Earlier validation of the shared canonical storage implementation: seven existing canonical-body tests passed. These do not substitute for the new integration regressions.

Passed: CARGO_BUILD_JOBS=1 cargo check --profile bounded-test -p tracedecay --no-default-features --lib --locked. This validates the daemon library wiring in the minimal feature configuration. Full production-feature builds and daemon runtime tests remain outside this check.

Two test-only integer inference/decoding issues found during the compiler gates were corrected, and the affected global-db and writer gates passed on rerun.

The new Rust runtime regressions have not executed. Test-binary code generation for the large asupersync dependency was killed by the 8 GiB environment limit in prior bounded attempts, including a serialized run. Compiler checks and direct SQLite probes are reported separately and are not Rust runtime-suite passes. Bazel was not run, per the requested Cargo validation workflow.

Rollout and limits

This PR remains draft until the new Rust runtime regressions can execute on a runner with enough memory. The code has not been deployed and no operator history or live database was modified.

Generic media/tool-payload encodings are unchanged. Restoration itself does not claim reclaimed disk bytes: old files await existing GC ownership checks and grace; physical database shrink depends on later checkpoint/compaction work. Production before/after disk savings have not been measured.

Implemented and reviewed with GPT-6 Astra Pro in ChatGPT.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 4aebbd3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Consolidated into #4742 (integrate/open-pull-requests) at the captured head shown in its description, as requested. Continue review and further fixes on #4742. This closure does not assert CI readiness; the replacement remains draft. Source branch retained.