Skip to content

fix(push): retain and retire capsule staging - #572

Closed
forhappy wants to merge 3 commits into
codex/capsule-delegated-recipesfrom
codex/capsule-snapshot-lifecycle
Closed

forhappy wants to merge 3 commits into
codex/capsule-delegated-recipesfrom
codex/capsule-snapshot-lifecycle

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Capsule push uploads and publishes selected pointers but does not retire their local staging owners. It also bypasses native-add intent recovery. A pointer committed before an interrupted native add can lose its local payload when the current Git index is restored to the prior state.

This change holds an exact staging snapshot through publication, retires captured owners only after confirmed remote commit, and preserves original leases on rejection, cancellation or failure. Sibling readers defer reclamation to maintenance. Native recovery checks local refs/reflogs before rolling back an old index state and creates canonical history ownership first. Git can return exit zero while omitting missing reflog tips, so recovery also requires a diagnostic-free walk before accepting absence. Protected finalize responses with coordinator metadata must attest Committed or Materialized before authorizing retirement.

Stacked on #570; draft while final CLI/large-file qualification remains pending. No merges requested.

Evidence and ownership

Surface Entry/caller Shared owner/callee Sibling contract
Capsule publication CLI and remote-helper capsule push StagingPushSnapshot; existing exact snapshot retirement and physical deletion Legacy push retains its existing retirement API
Interrupted native add Capsule push, native add open, legacy push git/index.rs recovery; staging history ownership Exclusive/shared recovery both preserve committed recipes before rollback
Protected finalization Crab Auth and managed finalize clients Shared validate_push_finalize_response Both reject pending, uploaded-only and aborted coordinator states
Existing behavior #570 capsule path; earlier CLI baseline Remote publication succeeds but owner/prepared payload remains Prior-index restoration previously deletes committed native payload

Is this the best fix? Ownership belongs in the existing staging machinery, while callers establish Git acceptance and remote commit. The guard and recovery use those boundaries instead of adding another staging format or payload store. Added implementation surface covers cancellation cleanup and history preservation; tests account for most of the growth. No dependency, lockfile or storage-format changes.

Validation

  • Six staging integration tests pass: snapshot drop/task abort, shared native ownership, deferred retirement, local journal failure, and prepared-only history preservation/reconstruction/physical cleanup.
  • Minimal and client-feature protected-push contract suites: 30 pass each.
  • Product recovery/capsule regressions: final-source run pending in the updated Linux/macOS/Windows CI selections. The local predecessor build was deliberately interrupted after the new scanner fix made its source obsolete.
  • Extracted exact scanner plus the canonical process owner, linked against the actual locked release error/dependency types: negative control reproduces unsafe acceptance of a missing reflog tip; patched probe retains the healthy result and rejects the incomplete walk. Helper/process source hashes are recorded.
  • Python harness syntax checks pass; Rust formatting and whitespace checks pass. Protocol workflow YAML parses, and its 16 evidence/compatibility helper tests pass.
  • CI now explicitly selects staging guards/history recovery, auth finalize validation, native intent recovery and capsule publication regressions. Linux runs the full capsule module; macOS/Windows run the lifecycle caller regression.
  • Current-source S3-compatible RustFS release canary at 7abb53e fails at git push --force origin main: the remote rejects it with stale info. This remains an unresolved correctness gate; no successful final-source full canary is claimed.
  • The previous CI run passes provider contracts and the initial native cutover push retirement check, then fails because its fault fixture deletes a retired manifest key. Updated missing/corrupt fixtures target the live v2 root and verify the fault before retaining the list/clone failure and integrity-error expectations. Selected corrected fixtures pass against the unchanged predecessor root reader: 23 commands/checks. The 11 source-isolation tests also pass.
  • Existing harness adds deterministic repeated/zero inputs, native-first/Git-first sequences, a local storage bound that rejects a second complete representation, and post-push ownership assertions for native/Git-helper flows. Reduced predecessor-binary runs (Git-first repeated and native-first zero; two 128-MiB files and four versions) each pass 71 commands and 53 checks, including all eight cold hashes and fsck. An older binary fails the storage bound after native-to-clean crossover (67,156,162 prepared bytes plus 68,224,532 segment bytes versus a 102,760,448-byte bound). Later inclusion of stream-temporary bytes and the full final-source 10-GiB matrix remain unqualified.
  • Independent frozen fix(push): admit delegated capsule recipes #570 10-GiB run: v1 Git-clean, v2 native and v3 Git-clean pushes pass, uploading 2,216,109, 2,133,870 and 2,214,948 incremental xorb bytes. Entropy replay completes: 52 commands and 41 checks, all eight historical 10-GiB file hashes after source/cache removal, independent clone, Git fsck and remote integrity with zero errors/repair failures, and unchanged binary. Report SHA256 e244d527e28b724df204e70abd4207141cba0ae52cbe271e7e7b891f611f4077. This is predecessor evidence, not validation of this PR's final binary.

Remaining gates

Final-source CLI/RustFS smoke, complete 10-GiB entropy/repeated/zero crossover and cold reconstruction, representative S3 cold/warm performance, and broad CI/cross-platform validation remain required before readiness. The architecture gate fails at its retired pack-install expectation: crab/src/read/mod.rs: pack object layout must delegate to crab-storage. Its baseline is not changed here.

The two requested large-file push audit Markdown/JSON files are excluded.

The eight stale double_must_use expectations introduced by #208, absent on origin/main, are removed in follow-up #573. That exact source passes strict staging/auth all-targets client-feature Clippy with -D warnings and no lint exceptions. The runtime worktree remains frozen.

Current-source local release proof (7abb53e): optimized minimal-feature build passes in 20m12s, binary SHA256 42a1aefb20f2aad9dbeac8a5c3839f7b64aaf099acc5b2a7c449381e46ca6d4b. Selected real RustFS smoke passes 73 commands and 55 checks: native add/push, Git-clean/helper push, fresh reconstruction and exact staging retirement, plus missing/corrupt canonical root refusal. This is selected-surface proof, not a successful complete canary. The real 10-GiB Git-first repeated-content/four-version run is active against this same frozen binary; no completed result is claimed yet. Force-push admission and visibility are addressed by draft follow-up #575, whose full CI remains pending.

Pin exact staging owners through remote publication. Preserve committed recipes before rolling back interrupted native index intents, and reject uncommitted protected finalize states before releasing local payloads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant