Integrate indexed catalogs, staging and lifecycle

crab/docs/architecture/git-protocol-v2.md

@@ -55,9 +55,8 @@ the invocation rather than allowing every protocol globally:
 git -c protocol.crab.allow=always clone --recurse-submodules crab://bucket/team/repo
 ```
 
-"All commands" is not a support claim. Date/ref-exclusion shallow selectors
-remain unimplemented; signed push certificates, push options intended for
-server-side hooks, and remote archive service need separate protocol contracts.
+"All commands" is not a support claim. Signed push certificates, push options
+intended for server-side hooks, and remote archive service need separate protocol contracts.
 Signed commits/tags are Git objects and are distinct from signed push requests.
 PRs, CI execution, and mandatory server-side hooks cannot execute inside a
 passive bucket. Clients with unrestricted bucket write credentials can bypass
@@ -76,7 +75,7 @@ select a v2 authority. The session supports:
 - `ls-refs` with ref prefixes, symrefs, peeled tags, unborn HEAD, and hidden
   refs;
 - `fetch` with wants, haves, done, tags, sideband/progress, shallow/deepen,
-  and `deepen-relative`;
+  `deepen-relative`, `deepen-since`, and `deepen-not`;
 - the bounded filter matrix: `blob:none`; `blob:limit=<n>[kmg]`; `tree:<depth>`;
   `object:type={tag,commit,tree,blob}`; `sparse:oid=<full SHA-1>`; and
   repeated filters or percent-encoded `combine:` intersections;
@@ -85,6 +84,12 @@ select a v2 authority. The session supports:
   required or assumed;
 - standard local Git pack installation on the v2 path.
 
+The line helper normalizes `--shallow-since` date expressions through Git's
+date parser before passing epoch seconds to the shared planner. Date and
+ref-exclusion selectors apply to requested tips as well as their parents.
+If no commits qualify, fetching fails before pack installation or shallow-file
+publication; it does not substitute a depth-one checkout.
+
 An empty manifest is a valid ref snapshot without a catalog. Empty clones and
 `ls-remote` complete through v2; clients requesting `unborn` receive the
 configured HEAD target unless hidden. Object wants against that empty snapshot
@@ -229,7 +234,7 @@ takeover.
 ## Explicitly unsupported
 
 The first profile rejects stateful `connect`, `packfile-uris`, `object-info`,
-`ref-in-want`, date/ref-exclusion shallow selectors, and other capabilities not
+`ref-in-want`, and other capabilities not
 listed above. A v2 session cannot fall back to the line protocol after the
 terminal handoff; it reports the protocol failure instead.
 
@@ -249,6 +254,13 @@ filter reads must leave canonical object counts and bytes unchanged; generated
 response-pack cache and coordination objects are accounted for separately. This
 is not a claim that first-read repair works with read-only object-store credentials.
 
+The add/commit/push RustFS smoke exercises the current physical fault boundary:
+loss of a published capsule and corruption confined to its embedded Git pack
+or index range. Each rejection probe uses an empty cache and requires no local
+HEAD or checked-out payload. It compares the remote root and ref-head bytes
+before and after the fault; discovery can itself reject a damaged capsule,
+so successful `ls-remote` is not the mutation oracle.
+
 Direct pushes and protected/service publication paths publish visibility
 before the corresponding ref or manifest commit. Protected receive extracts
 the closure from the already-verified materialization ODB before that workspace

crab/docs/architecture/indexed-remote-reads.md

@@ -69,7 +69,9 @@ counts contain no raw paths, OIDs, endpoints, content or credentials; separate
 oracle evidence contains the paths/OIDs needed to prove correctness.
 
 `/usr/bin/time` supplies process CPU and peak RSS. Those cover opening and both
-queries together; they are not per-query RSS. Index construction uses an explicit
+queries together; they are not per-query RSS. Catalog qualifier totals also
+include the complete inspection oracle, so they cannot establish selected-query
+memory bounds. Index construction uses an explicit
 `build-indexes` operation, outside measured reads, with its own requests, bytes,
 CPU and peak RSS. Read-only qualifications do not trigger maintenance.
 
@@ -2188,11 +2190,212 @@ two path heads, two leases, 65 payload records and 4,194,304 retained segment
 bytes remain. These receipts establish the oracle and staging component; they
 do not qualify the new integrated binary or catalog format.
 
-Combined-source CLI compilation and runtime qualification are still pending.
-Remaining catalog consumers include shared/protected preparation, pinned
-hydration/mount lookup and the current-state file lookup opener. Complete
+The combined release binary compiled and the focused RustFS smoke passed
+163 commands and 84 checks, including exact retirement, committed restaging
+before first push, both cold versions, force/ref updates and invalid roots.
+The integrated native-first entropy 10-GiB run passes 75 commands and 57 checks:
+all four pushes, exact retirement, eight cold historical SHA-256 checks, Git
+integrity and remote fsck with zero errors or repairs.
+Those receipts bind the earlier integrated source, not subsequent reader changes.
+
+Pinned hydration/mount lookup uses captured catalog pages in PR #581. PR #584
+extends this to current-state file lookup and shares ref ordering and captured
+visibility with Git readers. Metadata tests pass (458), as do all 47 read capsule-protocol tests after
+preserving corruption classification at the read boundary. CLI, runtime and
+retained-root lifecycle checks are required before readiness.
+A reproduced independent-ref fixture also requires complete inspection to
+validate dependencies after assembling the visible set, while publication-time
+delta validation remains strict.
+
+`qualify_remote ... <catalog-file-hash> catalog-current` measures the actual
+current-state lookup session. The argument uses Xet `MerkleHash::hex()` encoding,
+not the pointer wire format’s raw-byte hex. Hydration converts parsed pointer
+bytes with `MerkleHash::from(pointer.file_hash)` before lookup; the qualifier
+accepts that catalog identity directly. Opening, cold queries and warm queries are separate; the full
+inspection oracle runs after measured queries and explicit session close. This
+keeps unrelated catalog decoding out of cold-query preparation. Use a quiescent,
+dedicated fixture so the post-query full oracle has the same visible authority.
+Compare these
+receipts across fixed small edits and increasing distinct historical chunks,
+ref counts and uncompacted history; checkpoint row bounds alone are insufficient.
+
+Shared/protected preparation remains a separate integration boundary. Complete
 inspection remains appropriate for fsck, compaction and GC; foreground lookups
-must not inherit those scans. Carry forward applicable CLI range/cancellation
-contracts before final qualification. The complete integrated 10-GiB matrix,
-history scaling, lifecycle/migration, fuller blame and representative S3 gates
-remain open.
+must not inherit those scans. CLI snapshot reads still install all Git packs;
+carry forward the range/cancellation and cleanup contracts when removing that
+cost. The final combined 10-GiB matrix, history scaling, lifecycle/migration,
+fuller blame and representative S3 gates remain open.
+
+The GC regression passes with both existing cleanup tests (three tests total).
+It captures a 20,000-file checkpoint catalog without a file query, replaces it,
+and sweeps past the grace period. Actual orphan deletion is required; the old
+reader then fetches cold pages and resolves duplicated file requests, while a
+new session observes the replacement. Both checkpoints have a valid physical
+source descriptor/body. This is an in-memory metadata lifecycle proof, separate
+from payload hydration and concurrent real-store qualification.
+
+Current-state qualification against the four-version, two-file 10-GiB entropy
+history passes with the updated reader. Opening uses 28 backend operations and
+149,860 bytes. The duplicate-file cold query uses 497 range reads and 7,434,846
+bytes; the warm query repeats all 497 reads. Both results match full inspection,
+operations drain and no writes occur. These are one local debug-profile sample,
+not cloud latency or independent history-scaling proof. Repeated recipe metadata
+reads are a measured remaining performance gap, not a successful warm-cache gate.
+The first attempted receipt remains failed: it supplied pointer-wire hex where
+the qualifier requires the distinct catalog `MerkleHash::hex()` representation.
+
+A differential run of the same qualifier binary against a real 4-MiB file passes:
+five cold ranges / 3,755 bytes and zero warm backend operations. Combined with
+the unchanged captured lookup and the existing 256-page / 8-MiB FIFO page cache,
+this supports recipe-page eviction as the cause of repeated large-file warm
+reads. A bounded cache of already-verified file bindings is the next candidate;
+it must preserve the initial complete dependency proof, immutable capture,
+ordered duplicates/misses, error retry and shared close/cancellation contracts.
+
+The current reader's minimal CLI/NFS check and debug build pass. The force-push
+policy/visibility regression also passes after awaiting the now-async visibility
+API in its test caller; all six policy cases retain their assertions. Eleven
+Python fixture tests pass. The previously qualified staging-version assertion
+from the combined branch is carried forward with exact post-push retirement.
+
+The complete current-source RustFS CLI smoke remains failed (152 commands,
+61 checks). Its shallow-since case supplies an ISO date; the helper treats the
+remote-helper option as an integer wire timestamp and rejects it before clone.
+The harness still expects the earlier unsupported-option diagnostic. Git's
+remote-helper boundary passes date text through, while wire fetch uses a
+normalized timestamp. Fixing that adapter and qualifying dated boundaries
+against native Git are required; changing the expected message alone would
+hide the missing behavior. No full CLI E2E pass is claimed from this receipt.
+
+
+## Capsule chunk authority and exact staged reuse in progress
+
+PR #593 publishes rebuildable global chunk candidates after capsule ref
+publication. Each candidate names a source shard in the source repository's
+current authenticated catalog and a canonical origin receipt in the caller's
+trusted global scope. A physical xorb alone is insufficient: its catalog shard
+must retain the payload through the existing GC contract. Direct CLI publication
+and protected receive share the metadata encoder and writer. Protected receive's
+80 native-Git/in-memory tests pass; this is not real-S3 qualification.
+
+The follow-up stores each placement under its immutable digest as well as the
+replaceable chunk head. Push re-reads the exact add-time placement and origin
+IDs, validates source membership, and requires current provider identity even
+with warm payload caches. Its read-only proof session closes before publication
+opens the global writer. Verified source descriptors and original receipts stay
+caller-owned, with one retained-metadata byte allowance across source repositories.
+Conflicting descriptors or physical assertions fail explicitly.
+
+A regression measured one redundant full xorb GET after otherwise successful
+source/origin proof. Adopting the authenticated descriptor removes that GET:
+the first before/after pair fails at one versus zero body reads, then passes the
+unchanged zero-read bound. Final-source validation passes all 10 focused CLI/Xet tests, including partial
+add-time hints and mixed stale/fresh origin assertions. The cold source fixture
+uses native Git and real staging with an in-memory object store.
+Released v1.2.4 manifest receipts lack these capsule descriptors and still require
+full-body verification when imported into capsule publication.
+
+The dedicated cross-repository smoke oracle now requires one complete capsule
+source-proof event and its matching adoption event. It checks exact planned,
+indexed, matched, source-verified and payload-verified counts, with zero stale,
+mismatched or globally reclassified chunks. Five oracle tests pass; the original
+substring oracle rejected valid capsule proofs and accepted counters assembled
+from different events. This representation correction is not product E2E proof.
+Payload-only run tests pass (28), candidate/key-codec/writer tests pass (19),
+existing origin/range siblings pass (12), and strict minimal and remote-index
+metadata Clippy pass. The production CLI binary build also passes.
+
+The current-source RustFS cross-repository fixture passes 34 commands and 32
+checks at 4 MiB (`capsule-exact-proof-r2-c6a202ae`, code head `098859c02f1`).
+The source worktree is clean, and its recorded source hashes remain unchanged
+throughout the run. Consumer add retains neither segment nor prepared payload;
+push revalidates all 58 exact placements with zero stale, mismatched or globally
+reclassified chunks and uploads no new xorb. Add and push each read two xorb
+ranges totaling 2,658 bytes (footer and chunk metadata), with no full xorb GET.
+A separate empty clone cache then performs a real remote payload GET during
+hydration and produces the original SHA-256. This is one local debug-profile
+correctness fixture, not a latency or history-scaling qualification.
+
+The binary SHA-256 is
+`5002c1c59f8e2d6d3fa82abed1c5686c67d46feb93f975c0c1d76c1930f3dc70`;
+the report SHA-256 is
+`36d6ec1b744a2f5545a9a998710271bc2fa092ceadf756dd56c95acf75eeff2f`.
+
+The shared `crab-remote` preparation consumer, final 10-GiB qualification,
+history scaling, provider latency, lifecycle and full blame semantics remain
+open gates.
+
+
+## Shared capsule preparation in progress
+
+The shared `Prepared::upload_capsule` caller in HTTP receive now uses captured
+file, shard and xorb lookups instead of materializing unrelated historical
+catalogs. An ordinary Git upload with no new Crab pointer performs zero catalog
+reads. Selected pointer and prepared-content closures preserve size, membership
+and descriptor conflict checks. Capsule-specific implementation lives in
+`crates/crab-remote/src/prepare/capsule.rs`; its public API remains under
+`crab_remote::prepare`.
+
+Four regressions failed before the change and pass afterward: one redundant
+historical-catalog read for a Git-only upload; Xet-word-order formatting of a raw
+body digest; acceptance of rewritten xorb/shard scratch bodies after inspection;
+and no shared chunk candidate index after a successful native-content commit.
+The initial Git-only fixture sent an all-history pack containing an old pointer;
+its corrected incremental pack excludes the advertised tip before measuring.
+No weakened assertion or production failure is inferred from that fixture issue.
+
+Six public-API integration tests and strict all-target Clippy pass. The tests use
+native Git packs, real Xet artifacts, explicit publication leases, exact origin
+and source-anchor checks, complete file reconstruction, and a noncanonical index
+fault that must leave committed refs successful. The SlateDB test dependency uses
+the same pinned 0.15.0 release already used by metadata; the lockfile adds only
+that test dependency edge. Production dependency versions do not change.
+
+The dedicated RustFS public-API test passes for both newly created and occupied
+xorb keys. It uses native Git preparation, real S3 publication, cold source and
+origin-anchor checks, and complete file-hash reconstruction. The historical
+pointer integration also accepts the correct declared size and rejects a
+conflicting one. The final source passes strict all-target Clippy and the
+no-default-feature check.
+
+Eight HTTP receive/fault tests pass. Three dedicated real-RustFS cells also pass:
+the public native-content API, native Git push/fetch and browser editing over the
+HTTP listener, and the six receive-fault/restart scenarios. Source hashes remain
+unchanged throughout these runs. The API test-binary SHA-256 is
+`fe8a75004fd4ef7ec0946b999d32a829b83b28f94b51317a9c34e9990e9ebf5d`; the HTTP test-binary SHA-256 is
+`92318648278160dd57f96f35dbcffc01f44799a75a43b84442251965c07fef51`. These are small correctness fixtures, separate from the required
+large-file, growth and latency cells.
+
+The final combined 10-GiB matrix, unrelated-history/ref/frontier scaling, broader
+lifecycle/provider proof and fuller blame semantics still prevent completion.
+
+
+## Combined catalog/staging qualification and HTTP fault boundaries
+
+The combined revision carries selected publication, verified catalog bindings,
+source/browse retention, retained-source fsck, exact capsule chunk reuse and
+shared native-content preparation. Focused checks passed: 558 metadata tests,
+81 protected-receive tests, six shared-preparation tests, 242 staging tests,
+26 CLI integration tests, 57 CLI siblings, 20 origin/lease tests and eight HTTP
+receive/fault tests. Explicit live tests remain separately qualified; these
+counts do not establish the full release gate.
+
+The first combined RustFS native HTTP fixture returned 202 during its
+corrupt-descriptor case, where the corruption oracle required 503. Readiness
+can become visible before the publishing repair worker finishes. A diagnostic
+assertion with a one-second delayed generation-owner release response proved
+that the fixture could inject its next fault while that worker was active.
+The shared native HTTP fixture now awaits maintenance completion before each
+fault, reloads the current descriptor and checks its snapshot identity. All
+original status, error-code, recovery and Git assertions remain intact. The
+controlled probe then passed across ten delayed lease releases, and all eight
+HTTP unit/fault checks passed. This changes test synchronization, not the valid
+production distinction between incomplete readiness and corrupt metadata.
+
+Raw before/after evidence remains outside the checkout in
+`final-integration-http-metadata-probe-r4.json`,
+`final-integration-http-metadata-probe-r5.json` and
+`http-fault-boundary-after-r1.log`. Combined-source 10 GiB qualification,
+representative S3 latency, additional history/ref scaling and fuller Git blame
+semantics remain open gates. Earlier component receipts retain their original
+source identities and are not final combined-source proof.