feat(oplog): make a control v2 cut's pre-cut manifest durable - #1364
Merged
Merged
Conversation
…first tag The log at `log_id = 3` carries authority-inert bookkeeping artifacts and is named for that class; its op module, ops enum and ingest validator were named for the snapshot, the only tag defined so far. Rename `account::snapshot` to `account::annex`, `SnapshotOp` to `AnnexOp` and `validate_storable_snapshot_payload` to `validate_storable_annex_payload`. The snapshot-specific projection, selection, verification and authoring keep their names.
A v2 revocation names its evidence as a detached manifest, by digest. Nothing stored one and nothing handed one to the executor, so every revocation parked on `ParkCause::Manifest` and a pinned account applied v2 enrolments only. A manifest is now an ordinary annex-log entry at tag 1, carrying the planner's own canonical encoding verbatim — the digest a cut signed is the digest of the stored payload, so nothing can re-frame it out of reach. That needs no schema change and no wire change, and it replicates through the existing account-entry selectors. A pinned refold reads the held manifests back and hands them to the executor as the cut's evidence, which is what lets a revocation reach `Applied` and close its subject's roster seat. Three constraints shape it: - A manifest must be authored by the device that authors its cut. An unknown signer parks in `account_pre_verify`, which is capped per account and evicts oldest-first, and evidence a cut depends on for as long as it exists cannot live somewhere it can be evicted from. - One view is carried by one 64 KiB envelope, so a view names at most `MAX_VIEW_ENTRIES` (1,900) identities rather than the planner's bundle bound of 4,096. Splitting a view across entries is not the alternative: a cut names one digest over one payload. - A slice of the per-account candidate budget is reachable only by a manifest. Cut and manifest compete for the same grow-only budget and the cut can land first, so without the floor an exhausted account parks its revocations permanently, leaving the devices they name un-revoked. A manifest is deliberately not gated the way `usable_snapshots` gates a snapshot: its integrity is the digest the cut signed, not its carrier's live authority, and gating it would make a revoked author's manifest vanish and re-park a cut that had already applied. The planner's declared evidence budget and the candidate store's per-account budget are equal by design, which is what lets a refold hand the executor every row it holds; that equality is now asserted at compile time rather than left to a comment.
…he header production writes The view-entry ceiling test measured a hand-built header and asserted only an upper bound on the slack, so it could answer neither of the two questions that matter: whether a real chained manifest still signs, and whether the margin has been eaten. Both had to be settled by probing the code by hand. Measure the header `author_view_manifest_in_tx` actually writes, in both shapes it takes -- a device's first manifest and a chained one, whose `prev_hash` and grown `seq`/`auth_len` are what bind in production -- and assert the slack from both sides. The lower bound is the margin a new header field would have to eat before `MAX_VIEW_ENTRIES` became unsignable; a change that eats past it has to bring the bound down with it.
Contributor
SCIP oracle — resolution reportHeuristic→compiler edge resolution per corpus. Δ compares resolved-after to the
resolved = |
Contributor
|
Bencher Report| Project | rag-rat |
| Branch | feat/1311-manifest-durability |
| Testbed | hetzner-bigmem |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
- Instructions (instructions)
- LL Hits (hits)
- L1 Hits (hits)
- Total read+write (reads/writes)
- RAM Hits (hits)
- Estimated Cycles (cycles)
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | Estimated Cycles | cycles x 1e6 | Instructions | instructions x 1e6 | L1 Hits | hits x 1e6 | LL Hits | hits x 1e6 | RAM Hits | hits x 1e3 | Total read+write | reads/writes x 1e6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rag_pipeline::pipeline::index cargo_resolver:resolver_config() | 📈 view plot | 1,762.88 x 1e6 | 📈 view plot | 1,098.03 x 1e6 | 📈 view plot | 1,537.51 x 1e6 | 📈 view plot | 36.23 x 1e6 | 📈 view plot | 1,264.05 x 1e3 | 📈 view plot | 1,575.00 x 1e6 |
| rag_pipeline::pipeline::query_cold cargo_resolver:resolver_built_config() | 📈 view plot | 265.22 x 1e6 | 📈 view plot | 175.64 x 1e6 | 📈 view plot | 249.80 x 1e6 | 📈 view plot | 2.86 x 1e6 | 📈 view plot | 31.76 x 1e3 | 📈 view plot | 252.70 x 1e6 |
| rag_pipeline::pipeline::query_warm cargo_resolver:resolver_index() | 📈 view plot | 238.25 x 1e6 | 📈 view plot | 157.33 x 1e6 | 📈 view plot | 224.44 x 1e6 | 📈 view plot | 2.54 x 1e6 | 📈 view plot | 31.29 x 1e3 | 📈 view plot | 227.02 x 1e6 |
Contributor
|
| Project | rag-rat |
| Branch | feat/1311-manifest-durability |
| Testbed | hetzner-bigmem |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result seconds (s) (Result Δ%) | Upper Boundary seconds (s) (Limit %) |
|---|---|---|---|
| index_time/full_rebuild_cargo | 📈 view plot 🚷 view threshold | 6.52 s(+13.40%)Baseline: 5.75 s | 7.08 s (92.12%) |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…f formatting the outcome The capacity-floor assertions interpolated the `IngestOutcome` that `account_ingest` returned into their panic message. Formatting a returned value into a panic message is the shape a cleartext-logging scan reads as writing that value to a log, and an assertion does not need the value to say what happened: it needs to name which branch the ingest took. Map the outcome to a stable token through a closed match on the variants and assert on that. The failure still names the branch it got instead of the one it wanted, and a new variant breaks the match rather than going unnamed.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A control v2 revocation names its evidence as a detached manifest, by digest. Nothing stored a manifest and nothing handed one to the executor, so every revocation parked on
ParkCause::Manifestand a pinned account could apply enrolments and nothing else. This makes that evidence durable, which is what stands between the merged engine and a working upgrade path.A manifest is an ordinary annex entry
The annex log (log 3) already exists for signed, authority-inert artifacts, and its module header reserved tag 1 for the next one. A manifest takes it: same
account_ingestdriver, sameinsert_candidatewriter, sameaccount_entriestable. Zero DDL, zero wire change, and replication is inherited —account_entries_for_sync/_for_enrollmentselect every log, frames carry opaque bytes, and the session never inspects log id or entry type. An un-upgraded peer stores and relays a revocation's evidence it cannot itself apply, because an unknown annex tag is storable, relayed andRetainedUnfolded, and the closed-tag quarantine isCONTROL_LOG-scoped.The payload is the manifest CBOR verbatim, so
sha256(stored payload)is the digest the cut signed — on author, re-ingest and replication alike.execute_heldnow takes manifests andderive_pinned_projectionsuppliesheld_view_manifests(rows).The first commit renames
account::snapshottoaccount::annex: the module header already called it the annex log, and leaving a manifest insnapshot::opswould have been the misnomer becoming load-bearing. No shims — every call site migrated in the same change.Evidence is never gated on its carrier's authority
A snapshot is a claim, so
usable_snapshotsscoping it to a live incarnation is right. A manifest asserts nothing; its integrity is the digest the cut signed. Copying that gate would make a revoked author's manifest vanish and re-park a cut that already applied — un-revoking a device with the very revocation that removed its author. The manifest entry carries noparent_refand noauthority_ref, which removes the field a future reader would be tempted to gate on.A floor keeps evidence admissible
Cut and manifest compete for the same grow-only budget and the cut can land first, so an insider could fill the budget and leave honest manifests permanently unadmitted — parking those cuts forever, leaving their devices un-revoked, in a state capacity never drains out of.
A slice of the per-account budget is therefore reachable only by a manifest: 64 entries and 1 MiB, with ordinary traffic capped just below at 4032 and 15 MiB. The cost is real and deliberate — a 1.6% and 6.25% reduction for every account, pinned or not — and it is not silent:
candidate_capacity_headroomreports the ordinary caps, so enrollment preflight measures against the budget admission will actually apply.ENROLLMENT_HELD_ENTRY_HASHES_MAXstays at the full 4096, since manifests can still reach that ceiling and the total per-account count is unchanged.Using
account_candidate_reservationswas considered and rejected. That table charges a time-bounded obligation keyed by id, and the pruner deletes solely on expiry — so a non-expiring per-cut row keyed by the cited digest is unreapable by construction, and a few hundred cuts naming bogus digests would pin the byte budget permanently, converting insider-only starvation into an attacker-triggered one. A standing per-account row cannot express "this reserve belongs to one entry class", so admission would still need the class-aware branch. Gating the floor on pinned-ness fails for a different reason: the attack is to fill the budget before the pin lands.Bounds
MAX_ENTRIES/MAX_BYTESremain the bundle bounds and now carry compile-time assertions that they equal the store's per-account caps — the property that lets a driver pass the entire held set toplan_replaywithout selecting.MAX_VIEW_ENTRIES = 1_900is the new per-view bound, enforced on encode, on decode and in the length precheck, so a view between the two is neither authorable nor admittable.That bound is measured, not assumed: the payload is 64,661 bytes, leaving 670 bytes spare on a device's first manifest and 621 on a worst-case chained one. The test asserts both sides on the header production actually writes, with the lower bound on the chained shape — the upper bound alone would not have caught a bound raised toward the ceiling, because the origin shape still had margin when the chained one did not.
Verification
cargo nextest run -p rag-rat-oplog— 1153 passedcargo nextest run -p rag-rat-sync— 239 passed, 2 skippedcargo clippy-D warningson both crates,cargo check --workspace --all-targets,cargo +nightly fmt --check— cleanEight mutants, each confirmed to compile with the expected test failing by name: supplying no manifests; gating manifests on a live incarnation; removing the capacity floor; widening the per-view bound; re-framing the payload instead of carrying it verbatim; routing the manifest tag to the unknown-tag retain branch; a manifest signed by an uncertified device; and raising the per-view bound until the envelope margin vanishes. A ninth proves the bundle-bound assertion by failing the build.
Not done
Support gate semantics and their ~84 call sites are untouched — the gate flip is a later slice. Pin install stays test-only and no production cadence authors a manifest, matching
author_snapshot_in_tx. The same-author invariant (a manifest must come from its cut's device, or it parks in the evicting pre-verify queue and its cut can re-park after applying) is documented at the authoring seam but not mechanically enforced, because there is no cut-authoring path to bind it to yet;a_view_manifest_from_an_uncertified_device_is_only_parkedpins the consequence.Refs #1311.