perf(ledger): scope pruning to the validated checkpoint - #708
Closed
ScriptedAlchemy wants to merge 6 commits into
Closed
ScriptedAlchemy wants to merge 6 commits into
ScriptedAlchemy wants to merge 6 commits into
Conversation
The writer idempotency ledger was never pruned. Measured across seven stores it holds 1,149 MB of 3,112 MB total (36%), and 87% of three freshly indexed stores. Pruning by superseded incarnation would have reclaimed far more but is unsound: StoreIncarnationV1 is not monotonic — every non-daemon process derives it from random process-run bytes while the daemon uses a small counter, so a CLI attach can outrank a later daemon one and pruning by it would re-admit duplicate writes. This prunes only rows whose authority epoch trails the checkpoint epoch for that same incarnation, which never crosses a shard or incarnation boundary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
perf(observation) landed an import of tracedecay_runtime_core::background_cpu, but neither the module file nor its declaration was added, so the pushed integration branch does not compile: every branch cut from it fails on an unresolved import. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Retention ran as one shard-wide DELETE inside the user mutation's savepoint and the process's sole SQLite writer transaction. At the measured scale - 257,131 rows across seven stores - that monopolises admission, and an interruption or SQLITE_FULL rolls the checkpoint advance back with it, so every retry re-attempts the same delete and the new epoch may never commit. Each commit now removes at most one bounded batch, so the epoch advance commits regardless of how much backlog remains, and later commits at the standing epoch keep draining it until it converges. The delete also read each incarnation's raw scalar authority_epoch, though checkpoint::next decodes and validates only the incarnation being advanced. A neighbouring row whose scalar is corrupt - 999 while its watermark and receipt still encode 7 - would silently retire that incarnation's live receipts, re-admitting the duplicate writes they exist to stop. The candidate set is now scoped to the single incarnation whose checkpoint this commit validated, at that checkpoint's epoch; a neighbour's records are retired by its own commits. Pruning is still keyed on authority supersession, never on age and never on superseded incarnation: StoreIncarnationV1 is not monotonic, so a non-daemon attach can outrank a later daemon one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base landed its own copy of the retention rule, so both sides now prune superseded idempotency records. The two sides agree on the bounding scheme - same MAX_PRUNED_ROWS_PER_COMMIT name, same value of 256, same unconditional one-bounded-pass-per-commit placement after checkpoint persist - and that agreement is kept verbatim rather than replaced with a second competing scheme. What this branch still adds on top of that structure is the candidate scope. The base selects candidates by joining every checkpoint row and comparing each incarnation's raw scalar authority_epoch, but checkpoint::next decodes and validates only the incarnation being advanced. A neighbouring row whose scalar reads 999 while its watermark and receipt still encode 7 would therefore retire that incarnation's live receipts, and a lost receipt silently re-admits the duplicate write it exists to stop. The delete is now scoped to the single incarnation whose checkpoint this commit decoded, validated, and persisted, at that checkpoint's validated epoch; a neighbour's records are retired by its own commits under its own validated checkpoint. Resolution per file: - prune.rs: kept the base's bound and its per-commit placement, replaced the checkpoint join with the validated (incarnation, epoch) pair the caller already holds, and exposed MAX_PRUNED_ROWS_PER_COMMIT to the tests so they assert against the constant instead of a copy of 256. Ordering drops the two now-constant leading primary-key columns, which leaves the same key order over the same candidate set. - commit.rs: the call passes the persisted checkpoint instead of the bare shard id. Pruning still runs on every commit, so a backlog left by an earlier bounded pass still converges. - tests.rs: the base's two bounding tests are superseded by the same two tests deriving their expectations from the constant, so its copies are dropped rather than duplicated under both names. Both properties it covered are retained - one commit removes at most one bounded batch, and later commits at the standing epoch keep draining - alongside the new case that an unvalidated neighbouring checkpoint cannot retire its own receipts. Verified on the merge result: cargo check -p tracedecay-rusqlite-runtime --all-targets --locked ok cargo test -p tracedecay-rusqlite-runtime --locked 0 failed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…plan-reopened' into HEAD
|
Owner
Author
|
Closing as an exact no-op against PR 707. GitHub reports zero changed files, and the validated-incarnation pruning is already carried by 84df647 in the restored delivery tree. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopened against
codex/tracedecay-total-redesign-plan-reopened(prior PRs auto-closed when #421 merged and master was reverted).Base already carries the 256-row bound; this keeps that verbatim and adds only what remained: the delete is scoped to the validated
(incarnation, epoch)pair instead of joining every checkpoint row and trusting each one's raw scalarauthority_epoch.checkpoint::nextvalidates only the incarnation it advances, so a corrupt neighbour reading 999 while its watermark encodes 7 could retire that incarnation's live receipts — and a lost receipt silently re-admits the duplicate write it exists to stop.328 tests pass, 0 failures. Superseded-incarnation pruning stays out:
StoreIncarnationV1is non-monotonic.